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PREFACE 


In  the  fall  of  1989  we  were  approached  by  John  Hollerbach,  a  member  of  the  Engineering 
Conferences  Advisory  Committee  of  the  Engineering  Foundation,  and  asked  if  we  would  consider 
organizing  a  meeting  around  the  general  theme  of  teleoperation.  After  a  number  of  discussions  with 
a  variety  of  people  at  MIT  and  elsewhere,  we  decided  that  it  was  time  to  bring  together  in  an 
informal  setting  two  research  communities  that  are  working  on  distinct  but  closely  related  types  of 
systems:  teleoperators  and  virtual  environments. 

Accordingly,  we  invited  the  best  people  available  in  these  two  emerging  fields.  We  were 
extremely  pleased  that,  with  relatively  few  exceptions,  all  people  contacted  were  enthusiastic  about 
attending  the  proposed  meeting. 

We  decided  to  have  three  kinds  of  sessions:  invited-paper  sessions,  panel  discussions,  and  poster 
sessions.  The  invited  papers  were  divided  into  papers  providing  general  background  on  sensorimotor 
performance  and  human-machine  interfaces,  papers  on  current  problems  and  progress  in 
teleoperation,  and  the  same  in  virtual  environments.  Also,  the  panel  discussions  were  subdivided  into 
three  problem  areas:  (1)  telepresence  and  sensorimotor  adaptation;  (2)  performance  measurement 
and  evaluation  methods;  and  (3)  design  principles  and  predictive  models.  A  mini  seminar  on 
teleoperators  with  micro  slave  robots  was  also  included. 

In  order  not  to  burden  the  people  making  presentations,  we  did  not  require  preparation  of  written 
papers.  Accordingly,  this  report  in  no  sense  constitutes  a  proceedings.  It  is,  rather,  a  collection  of 
conference -related  fragments,  highly  non-uniform  and  totally  unedited,  that  we  hope  will  give 
people,  particularly  those  who  could  not  attend,  a  gliippse  of  the  potential  of  these  fields  and  how 
they  interplay.  We  have  included  titles  of  presentations  even  when  no  written  material  was  made 
available  (either  because  no  written  material  was  submitted  or  because  the  presentation  was 
withdrawn).  Interested  readers  should  contact  the  authors  directly  to  obtain  further  information  on 
these  items.  It  is  expected  that  a  book  on  this  topic,  nourished  by  this  conference,  will  be  published 
in  the  not-too-distant  future. 

We  are  grateful  to  Dr.  Melvin  Montemerlo  of  NASA  Headquarters,  Dr.  Michael  Shafto  of  NASA 
Ames  Research  Center,  Dr.  Harold  Hawkins  of  ONR,  and  Dr.  Daniel  Ling  of  IBM  for  their  generous 
support  of  our  meeting.  We  are  also  grateful  to  Dr.  Stephen  Ellis  for  his  help  in  preparing  this  NASA 
publication. 


Nat  Durlach  and  Tom  Sheridan 
Conference  Chairpersons 
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INTRODUCTION 


In  a  teleoperator  system  (see  the  aecompanyino  figure),  the  human  operator  senses,  moves 
within,  and  operates  upon  a  remote  or  hazardous  environment  by  means  of  a  slave  mechanism  (a 
mechanism  often  referred  to  as  a  teleoperator)., 'Signals  in  this  environment  are  sensed  by  devices  on 
the  slave  mechanism  and  communicated  back  to  a  human-machine  interface  and  displayed  to  the 
human  operator;  the  human  operator’s  responses  are  sensed  by  devices  at  the  interface  and  commu¬ 
nicated  back  to  the  slave  mechanism  and  used  to  control  it.  Teleoperators  have  been  in  use  since  the 
first  nuclear  laboratory  experiments  in  the  late  1940’s.  Teleoperation  in  the  deep  ocean,  in  space,  in 
mines,  and  other  remote,  often  hazardous,  environments  soon  followed.  Applications  in  medicine, 
involving  exploration  and  operation  within  the  human  body,  are  now  receiving  increased  attention. 
Early  teleoperators  were  directly  controlled  by  the  operator  in  position  (using  isomorphic  master- 
slave  manipulators)  or  in  rate  (where  joystick  position  determined  manipulator  or  vehicle  rate).  More 
recently,  computers  have  intervened,  and  the  operator  has  become  a  supervisor,  issuing  subtask  goal 
and  procedural  commands  and  letting  the  teleoperator’s  own  computer  then  implement  those 
instructions  by  closing  a  control  loop  through  its  own  sensors  and  actuators.  We  term  this  more 
sophisticated  teleoperator  a  telerobot  and  the  mode  of  control  supervisory  control.  General  discus¬ 
sions  of  teleoperation  are  available  in  Johnsen  and  Corliss,  1971;  Vertut  and  Coiffet,  1986;  and 
Sheridan,  1989. 

In  a  virtual-environment  system  (again  see  the  accompanying  figure),  the  interactive  human- 
machine  interface  is  retained  but  the  slave  mechanism  and  its  environment  are  replaced  by  a  com¬ 
puter  simulation.  Video  is  replaced  by  computer  graphics.  The  auditory  and  force  sensations 
imparted  to  the  human  operator  are  similarly  computer-generated.^Since  the  rules  governing  the 
behavior  of  the  simulation  are  under  the  control  of  the  system  designer,  the  operator  in  such  systems 
can  be  presented  with  totally  artificial  worlds  (c.g..  having  new  physical  Iaws).^n  contrast  to  a  tele¬ 
operator  system,  where  the  purpose  is  to  extend  the  operator’s  sensorimotor  system  in  a  manner  that 
facilitates  exploration  and  manipulation  of  the  physical  environment,  in  a  virtual-environment  sys¬ 
tem,  the  purpose  is  to  train,  inform,  alter,  or  study  the  human  operator  to  modify  the  state  of  the 
computer  and  the  information  environment.  A  major  application  in  which  the  human  operator  is  the 
target  (because  the  goal  is  training)  is  that  of  flight  simulation.  Although  flight  simulators  have  been 
around  for  more  than  a  decade,  they  had  little  impact  outside  aviation  presumably  because  the  appli¬ 
cation  was  so  specialized  and  so  expensive.  Clearly,  what  has  created  the  recent  flurry  of  interest  in 
virtual  environments  is  the  great  reduction  in  cost  and  increase  in  power  of  computer-graphic  work¬ 
stations,  augmented,  of  course  by  miniaturization  of  sensors  and  display  devices.  Now  some  of  the 
main  applications  are  occurring  in  the  entertainment  industry  (e.g.,  video  games).  Soon,  we  expect, 
there  will  be  an  increasing  emphasis  on  designing  virtual-environment  systems  specifically  for  sci¬ 
entific  study  of  sensorimotor  performance  and  for  behavior  modification  by  biofeedback.  Applica¬ 
tions  directed  more  towards  changing  the  information  environment  than  the  human  operato.  involve 
use  of  the  system  for  scientific  visualization  and  theoretical  modelling  (as  in  work  on  molecular 
docking).  Also,  virtual  environment  systems  are  now  being  incorporated  within  teleoperator  systems 
in  cases  for  which  planning  modes  or  predictive  model  outputs  are  required.  The  idea  in  this  case  is 
for  the  operator  to  stee  the  teleoperator  through  ‘■imulated  operations  presented  computer- 
graphically  just  before  performing  the  real  operations.  Alternatively,  computer-graphic  predictions  of 
trajectory  (bases  on  past  and  present  control  inputs)  can  be  displayed  automatically  while  the 


operator  fliis  the  vehicle  or  guides  the  manipulator .  Such  model  based  predictions  can  then  be 
superposed  on  the  teleoperation  video.  General  background  on  virtual-environments  can  be  found  in 
Sutherland,  1965;  Krueger,  1983;  Bolt,  1984;  Haber,  1986;  Foley,  1987;  Fisher  era/.,  1988;  Chung 
et  al.,  1989;  Lanier,  1990;  and  Brooks  et  aL,  1990. 

In  either  a  teleoperator  or  virtual-environment  system,  when  the  sensory  feedback  can  be  made 
sufficiently  compelling  (through  the  use  of  multimodality  displays  in  which  the  output  is  influenced 
by  the  motor  behavior  of  the  human  operator)  to  make  the  operator  feel  present  at  the  remote  or 
imaginary  site,  the  system  is  said  to  have  telepresence.  The  extent  to  which  telepresence  is 
important  for  achieving  good  performance  is  itself  an  interesting  research  topic. 

In  both  teleoperator  and  virtual-environment  systems,  the  human  operator  is  effectively  trans¬ 
formed  into  a  new  creature  that  is  present  in  a  new  interactive  environment.  Also,  in  both  cases  the 
characteristics  of  the  proposed  transformation,  as  well  as  the  sense  of  presence  (or  telepresence)  in 
the  new  environment,  are  strongly  influenced  by  the  human-machine  interface.  Although  there  are 
some  significant  differences  in  the  interface  issues  associated  with  the  two  types  of  systems  (for 
example,  in  a  teleoperator  system  it  may  be  impossible  to  code  certain  targets  independently  because 
of  inadequate  resolution  in  the  sensory  system  of  the  slave  mechanism),  there  is  clearly  much  in 
common.  It  is  these  common  elements,  namely,  the  transformed  presence  and  the  human-machine 
interface,  that  constitute  the  primary  focus  of  this  conference. 


Nat  Durlach  and  Tom  Sheridan 
Conference  Chairpersons 
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BACKGROUND  ON  SENSORIMOTOR 
PERFORMANCE  AND  HUMAN-MACHINE 

INTERFACES 


VISUAL  INTERFACES:  THE  HUMAN  PERCEIVER 


Richard  M.  Held 

Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts 


I  was  asked  to  speak  about  how  knowledge  of  visual  perception  could  be  applied  to  achieve  tele¬ 
operation  and  virtual  worlds.  After  a  little  thought  I  decided  that  reversing  subject  and  object  makes 
the  question  more  interesting  and  provocative.  The  pract  ce  of  teleoperaiion  and  virtual  world  tech¬ 
nology  challenges  our  understanding  of  perception,  and  in  the  following  I  try  to  explain  what  I  mean. 
But  ^:rst  let  me  say  that  vision  science  incorporates  a  vast  amount  of  useful  knowledge  about  the 
capabilities  of  the  visual  system.  Fortunately,  most  of  it  is  well-codified  in  the  handbooks,  of  which 
the  most  recent  and  complete  is  Boff,  Kaufman,  and  Thomas’s  Handbook  of  Visual  Perception  and 
Performance  r:'J89).  Unfortunately,  this  codificu  icnowledge  does  not  prepare  us  for  the  surprises 
attendant  on  the  sensory  and  motor  transformations  effected  by  the  new  technologies.  Instead,  the 
surprises  refer  us  to  much  less  well  understood  areas.  Hence  the  challenge  to  perceptual  science  from 
technology. 

World  Regularities  and  the  Habitual 


New  technologies  make  it  easy  to  transform  spatially,  temporally,  and  dynamically  the  relations 
among  sensory  inputs  and  motor  outputs.  For  example,  nothing  could  be  simpler  than  making  your 
hand  appear  to  reach  to  the  left  when  you  intended  it  to  reach  to  the  right,  tc  make  you  step  up  when 
you  should  step  down,  make  the  usually  stable  visible  world  appear  to  move  as  you  move,  and  so  on. 
The  consequences  can  be  errors  in  performance,  disorientation,  and  a  form  of  malaise  like  motion 
sickness.  These  examples  make  us  realize  to  what  extent  our  perceptions  are  dependent  upon  the 
regularities  and  habitual  circumstances  of  our  normal  world  and  the  dynamics  of  our  actions  in  it. 

We  are  preadapted  to  these  ordered  states  in  multiple  ways  that  are  revealed  by  the  effects  of  apply¬ 
ing  the  new  technologies  to  the  human-machine  interface.  Compared  to  our  knowledge  of  basic 
visual  processes,  our  understanding  of  these  orderings  is  limited  but  not  insignificant.  Some  of  it  has 
been  relegated  to  the  fringes  of  the  field  as  aberrant  and  bizarre  forms  of  perception.  The  followmg 
table  enumerates  several  types  of  transformation  and  examples. 


Type 

Spatial  Incongruity 
Temporal  Asynchrony 
Altered  Dynamics 
Deprivation 


Examples 

Japanese  Illusion 
Sensory  Rearrangement 

Phase  Shift 
Tim.e  Delay 

Microgravity 
Force  Amplification 

Blank  Visual  Field 
High  Noise  Level 
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Some  of  these  non- habitual  conditions  may  cause  initial  errors  and  disorientations.  Some  show 
adaptation  during  continued  exposure  n  the  form  of  error  reduction,  better  orientation,  and  reduction 
of  motion  sickness.  Adaptation  requires  maintenance  of  a  transformed  set  of  regularities.  If  the  trans¬ 
forms  are  complex  and/or  if  the  coupling  is  noisy,  adaptation  may  be  either  very  slow  or  not  possible 
at  all.  However  useful  adaptation  may  be,  it  does  take  time  and  represents  an  additional  load  on  the 
processing  capacity  of  the  system,  sometimes  called  a  cognitive  load. 

Telepresence  and  Presence 

A  particularly  intriguing  problem,  related  to  the  above-mentioned  considerations,  is  the  creation 
of  telepresence.  It  raises  the  question  of  what  conditions  specify  the  sense  of  presence  — a  question 
that  seems  at  first  blush  so  obvious  as  to  require  no  thought.  It  is  the  first  axiom  of  perceiving, 
namely,  that  the  observer  be  there  to  perceive.  It  implies  a  here  and  now  spatial  juxtaposition 
(confrontation)  of  object  and  observer.  We  can  begin  to  identify  conditions  for  the  sense  of  presence 
as: 

1 .  Independent  perception  of  an  observer  and  an  object  external  to  the  observer. 

2.  Covariation  of  sensed  observer  movement  independent  of  the  sensed  object  yields  the  sense 
of  an  object  external  to  the  observer. 

3.  Observer  has  access  to  the  sensed  object  at  will. 

Achievement  of  these  conditions  is,  in  turn,  related  to  the  regularities  and  habitual  circumstances 
discussed  above. 
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AUDITORY  INTERFACES:  THE  HUMAN  PERCEIVER 


H.  Steven  Colburn 
Boston  University 
Boston,  Massachusetts 


This  talk  is  a  brief  introduction  to  the  basic  auditory  abilities  of  the  human  perceiver  with 
particular  attention  toward  issues  that  may  be  important  for  the  design  of  auditory  interfaces.  The 
importance  of  appropriate  auditory  inputs  to  observers  with  normal  hearing  is  probably  related  to  the 
role  of  hearing  as  an  omnidirectional,  early  warning  system  and  to  its  role  as  the  primary  vehicle  for 
communication  of  strong  personal  feelings. 

Several  basic  properties  of  the  human  auditory  perceiver  should  be  kept  in  mind  when  designing 
interfaces.  The  range  and  resolution  of  the  perceiver  is  impressive  in  terms  of  frequency  (three 
decades  with  sensitivity  to  changes  of  0.2%)  and  intensity  (110  decibels  with  sensitivity  to  changes 
of  a  fraction  of  a  decibel  at  almost  all  levels)  so  that  the  standards  for  clean  sounds  are  quite  high.  On 
the  other  hand,  our  memory  for  absolute  stimuli  is  much  more  limited,  particularly  with  respect  to 
intensity.  In  an  identification  paradigm,  perceptual  systems  are  generally  limited  to  about  7  ±  2 
intensity  levels,  so  that  the  ability  to  identify  is  much  more  limited  than  the  ability  to  differentiate. 
The  notion  of  the  critical  bandwidth  is  also  important  in  understanding  auditory  phenomena  and 
abilities.  In  its  crudest  form,  the  critical  band  notion,  which  is  consistent  with  a  wide  variety  of  data, 
is  that  the  auditory  system  behaves  as  if  it  contained  peripheral  filters  with  bandwidths  of  about  one- 
tenth  of  an  octave. 

The  issues  that  arise  in  the  design  of  auditory  interfaces  are  different  according  to  whether  the 
interface  is  an  isomorphic  representation  of  the  acoustic  environment  with  natural  sounds  and  trans¬ 
formations  or  a  non-isomorphic  mapping  from  fundamentally  different  inputs.  In  the  completely 
isomorphic  case,  the  problem  is  conceptually  straightforward  but  very  difficult  to  realize.  Although 
one  simply  has  to  recreate  the  appropriate  acoustic  stimuli  (from  a  robot  or  from  a  simulation)  at  the 
ears  of  the  human  listener,  there  are  substantial  engineering  challenges  in  reproducing  or  creating  the 
appropriate  sounds.  In  addition  to  the  obvious  requirements  for  excellent  fidelity  in  the  acoustic  sys¬ 
tem,  the  echo  and  reverberation  effects  on  each  sound  must  be  appropriate  and,  critically,  the  motion 
of  the  listener’s  head  must  be  coupled  to  the  robot  or  program.  The  complexity  of  the  problem  can  be 
illustrated  by  thinking  of  the  effect  on  the  sounds  received  at  the  ears  by  several  sound  sources  in  a 
normal  environment.  In  this  case,  as  the  head  moves,  the  “head-related  transfer  function”  of  each 
source  changes  in  a  way  that  can  only  be  specified  with  knowledge  of  the  position  of  the  source  and 
the  position  and  orientation  of  the  head.  The  processing  of  the  human  perceiver  imposes  important 
constraints  on  the  quality  of  the  reproduced  sound  signal. 

In  the  simplest  non-isomorphic  cases,  such  as  a  scaled  robot  or  a  normal-sized  robot  in  an  envi¬ 
ronment  with  acoustic  properties  different  than  air  (e.g.,  a  helium  rich  environment),  there  are  differ¬ 
ent  problems,  including  the  translations  of  signals  to  signals  appropriate  for  human  listeners.  For 
example,  a  very  small  robot  would  have  very  small  interaural  differences  that  could  be  essentially 
undetectable  to  the  human  perceiver  with  an  associated  loss  of  ability  to  judge  the  azimuth  of  a 
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sound  source.  If  these  signals  are  processed  to  provide  an  appropriate  stimulus  on  a  human  scale,  the 
influence  of  the  environment  on  the  signals  would  have  to  be  transformed  and  this  would  require 
significant  knowledge  of  an  environment  that  may  be  unfamiliar  to  the  robot  or  its  controller.  The 
transformation  of  the  received  stimuli  to  provide  appropriate  acoustic  inputs  for  a  natural  (human 
sized)  situation  requires  a  relatively  full  knowledge  of  the  sound  field. 

In  the  case  of  more  extreme  deviations  from  a  humanoid  robot  or  simulation,  the  issues  become 
conceptually  difficult  to  think  about.  For  example,  if  the  robot  has  multiple  ears  so  that  a  more 
sophisticated  picture  of  the  acoustic  environment  could  be  computed  by  the  interface,  the  mapping  of 
the  signals  or  the  infoimation  to  the  two  ears  of  the  human  perceiver  is  difficult  to  design.  Either 
some  of  the  information  will  be  neglected  or  we  must  learn  how  to  increase  the  presentation  of 
information  to  the  human,  possible  by  parsing  the  acoustic  field  and  presenting  multiple  acoustic 
objects.  This  is  a  higher  level  recoding  problem,  and  depends  on  techniques  that  have  not  yet  been 
developed.  Some  of  the  attributes  that  result  in  the  perception  of  separate  acoustic  objects  are  known 
and  include  separate  spatial  location,  consistent  modulation  envelopes  in  frequency  and  amplitude, 
consistent  harmonic  structure,  and  others. 

In  some  cases,  if  a  stimulus  is  to  be  perceived  as  being  generated  by  the  listener,  then  there  are 
special  problems  due  to  the  fact  that  the  sound  is  being  received  over  two  pathways,  by  the  air-con¬ 
duction  pathway  and  by  the  bone-conduction  pathway.  The  air-conducted  sound  can  be  monitored 
and  transformed  appropriately  by  the  interface,  but  the  bone-conduction  pathway  is  difficult  to  can¬ 
cel  so  that  the  stimulus  perceived  by  the  subject  is  a  combination  of  the  bone-conducted  sound  and 
the  air-conducted  sound  provided  by  the  interface.  Learning  would  be  a  problem  here  if  the  new 
environment  was  trying  to  simulate  an  environment  with  light  or  heavy  gases  for  example.  In  this 
ca.se.  the  auditory  stimulus  would  be  a  combination  of  the  higher-pitched  sound  and  the  normal- 
pitched  sound. 

There  are  many  ca.ses  in  which  unnatural  stimuli  are  more  effective  than  naturally  occurring 
stimuli.  [The  argument  that  the  human  sensori-motor  system  is  optimized  by  evolution  (often  applied 
to  speech  signals)  cannot  be  applied  in  many  cases  because  the  constraints  of  the  system  are  so 
complex  and  even  difficult  to  know.  In  addition  to  questions  of  acoustics,  for  example,  one  can  ask 
how  important  is  the  eating  function  to  the  design  of  the  mouth  or  the  necessity  of  being  bom  to  the 
size  of  the  head?]  When  there  is  a  well-defined  task  to  be  performed,  it  is  often  advantageous  to 
provide  non-natural  proce.ssing  of  the  stimuli.  For  example,  angle  resolution  near  the  midline  can  be 
improved  significantly  by  presenting  nonlinearly  processed  stimuli  that  would  increase  the  interaural 
differences  even  though  some  distortion  would  be  added  that  could  be  important  for  some  stimuli. 

On  the  other  hand,  when  the  tasks  are  extremely  varied  or  unpredictable,  there  are  significant  advan¬ 
tages  to  using  natural  stimuli  and  allowing  the  experience  of  the  listener  to  be  used  to  primary  advan¬ 
tage  with  little  training  required  for  many  ta.sks.  In  addition,  there  are  situations  in  which  an  unnatu¬ 
ral  stimulus  may  be  advantageous  such  as  the  design  of  hearing  aids  with  expanded  ranges  of  levels 
and  frequencies  or  in  the  design  of  auditory  di.splays  for  non-auditory  information. 
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TACTUAL  INTERFACES:  THE  HUMAN  PERCEIVER 


M.  A.  Srinivasan 

Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts 


Increasingly  complex  human-machine  interactions,  such  as  in  teleoperation  or  in  virtual  envi¬ 
ronments,  have  necessitated  the  optimal  use  of  the  human  tactual  channel  for  information  transfer. 
This  need  leads  to  a  demand  for  a  basic  understanding  of  how  the  human  tactual  system  works,  so 
that  the  tactual  interface  between  the  human  and  the  machine  can  receive  the  command  signals  from 
the  human,  as  well  as  display  the  information  to  the  human,  in  a  manner  that  appears  natural  to  the 
human.  The  tactual  information  consists  of  two  components:  (1)  contact  information  which  specifies 
the  nature  of  direct  contact  with  the  object;  (2)  kinesthetic  information  which  refers  to  the  position 
and  motion  of  the  limbs.  In  this  talk,  we  shall  mostly  be  concerned  with  contact  information. 

A  wide  variety  of  tactile  perceptions  that  we  distinguish  can  be  thought  of  as  combinations  of 
two  classes  of  “primitives”:  mode  of  contact  (static  indentation,  relative  tangential  motion  or  slip, 
and  vibration)  and  object  properties  (surface  microtexture,  shape,  and  compliance).  It  is  plausible 
that  by  understanding  the  peripheral  and  central  processing  involved  in  each  pairwise  combination 
among  the  primitive  classes,  the  processing  of  more  complex  natural  stimuli  can  be  unravelled.  Our 
knowledge  of  cutaneous  information  processing  is  mostly  based  on  human  psychophysics  and  elec- 
trophysiological  recordings  from  monkey  afferent  fibers.  These  fibers  are  of  four  types:  Two  types 
of  slowly  adapting  fibers  (SA I  and  II)  that  are  associated  with  Merkel  cells  and  Ruffmi  endings,  and 
are  responsive  both  when  an  object  in  contact  is  moving  against  the  skin  as  well  as  under  steady 
indentations;  Two  types  of  rapidly  adapting  fibers  which  respond  only  when  the  skin  is  moving,  one 
type  (RAs)  terminating  in  Meissner  corpuscles,  and  the  other  (PCs)  in  Pacinian  corpuscles.  When  a 
probe  indenting  the  skin  is  vibrated,  lowest  response  threshold  amplitudes  for  SAs  are  at  frequencies 
of  0-10  Hz,  for  RAs  at  20-50  Hz,  and  for  PCs  at  l(X)-300  Hz.  Summarized  below  are  some  of  the 
known  results  on  the  detection  of  slip  and  microtexture,  shape,  and  compliance. 

Humans  cannot  detect  the  slip  of  a  smooth  glass  plate  on  the  skin;  existence  of  detectable  fea¬ 
tures  on  the  surface  is  necessary.  However,  surprisingly  small  features  (for  example,  2  microns  high 
dot  causes  RAs  to  respond  and  0.06  microns  high  grating  causes  PCs  to  respond)  on  smooth  surfaces 
are  detected  by  humans  and  lead  to  the  detection  of  slip  of  these  surfaces,  with  the  geometry  of  the 
microfeatures  governing  the  associated  neural  codes.  The  division  of  labor  among  the  different  types 
of  fiber  populations  in  signaling  the  different  events  on  the  skin  is  clear-cut:  SAs  signal  the  direction 
of  skin  stretch  and  hence  the  direction  of  impending  slip;  RAs  and  PCs  signal  the  occurrence  of  slip 
with  spatiotemporal  or  intensive  codes,  depending  on  whether  the  microfeature  is  a  local  one  on  a 
smooth  background,  or  is  distributed  on  the  surface,  respectively.  When  the  surface  features  are  of 
sizes  greater  than  the  response  thresholds  of  ail  the  receptors,  redundant  spatiotemporal  and  intensive 
information  from  all  three  afferent  fiber  types  is  available  for  the  detection  of  slip. 

Among  the  different  possible  geometric  representations  of  the  shape  of  objects,  the  intrinsic 
description,  i.e.,  the  surface  curvature  as  a  function  of  the  distance  along  the  surface,  seems  to  be 
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relevant  for  tactile  sensing.  Recordings  of  afferent  responses  to  diverse  shapes  show  that  the  depth  of 
indentation  and  the  change  in  cuiwature  of  the  skin  surface  are  represented  in  SA  responses;  in  addi¬ 
tion,  the  velocity  and  the  rate  of  change  in  skin  surface  curvature  are  represented  in  both  SA  and  RA 
responses.  The  primary  reason  for  such  neural  encoding  is  the  form  of  the  spatial  variation  of  the 
pressure  imposed  by  the  object  on  the  skin  surface:  Pressure  peaks  occur  where  the  depths  of  inden¬ 
tation  and/or  changes  in  the  skin  surface  curvature  are  high.  The  skin  effectiv  y  acts  as  a  low-pass 
filter  in  transmitting  the  mechanical  signals,  and  the  mechanoreceptors  respond  to  the  blurred  ver¬ 
sions  of  the  surface  pressure  distribution,  thus  encoding  the  shape  of  the  object  in  terms  of  its  surface 
curvatures. 

Compliant  objects  can  be  of  two  types:  (1)  those  with  a  rigid  surface  (such  as  a  piano  key); 

(2)  those  with  a  deformable  surface  (such  as  cheese).  By  conducting  psychophysical  experiments 
under  both  active  and  passive  touch,  with  or  without  local  anesthesia  that  blocked  the  contact  infor¬ 
mation,  the  following  conclusions  were  reached.  Humans  are  very  good  at  discriminating  softness  of 
objects  with  deformable  surfaces,  and  the  discrimination  is  based  on  contact  information.  For  com¬ 
pliant  objects  with  rigid  surfaces,  discrimination  of  compliance  is  reliable  only  under  active  touch, 
when  both  contact  and  kinesthetic  information  is  available,  and  the  discriminability  is  poorer  than 
that  for  objects  with  deformable  surfaces.  Preliminary  neurophysiological  data  show  that  mostly  SA 
and  to  a  lesser  extent  RA  responses  provide  the  basis  for  our  ability  to  discriminate  compliance. 
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THE  DISPLAY  OF  TACTILE  INFORMATION 


Carl  E.  Sherrick 
Princeton  University 
Princeton,  New  Jersey 


There  are  a  number  of  examples  of  natural  tactile  displays  that  can  give  us  some  insights  about 
the  soUd  geometry  of  touch,  and  recent  experimental  work  on  the  subject  has  extended  our  thinking 
considerably.  The  concern  of  this  conference  is,  however,  more  with  synthetic  or  artificial  displays 
for  the  production  of  a  virtual  environment.  The  majority  of  the  talk  will  dwell  on  the  features  of 
synthetic  displays  that  have  enjoyed  some  success  in  one  of  two  enterprises:  The  study  of  the  spatio- 
temporal  dimensions  of  stimuli  that  afford  accurate  and  rapid  processing  of  environmental  informa¬ 
tion,  or  the  use  of  displays  in  the  design  of  sensory  aids  for  disabled  persons. 

Examples  of  synthetic  displays  are  given,  with  some  discussion  of  their  application,  success,  and 
reasons  for  their  failure,  i.e.,  their  current  disuse.  In  the  past,  displays  have  been  mainly  of  the  type 
called  diffuse,  i.e.,  spread  over  a  large  area  of  the  body.  In  more  recent  years,  the  improvements  in 
electronic  technology  have  made  possible  the  production  of  more  and  more  dense  arrays  of  tactile 
stimulators,  some  of  which,  like  the  Optacon  of  Telesensory  Systems,  Inc.,  were  developed  for 
applied  purposes,  and  now  are  used  in  basic  research  as  well. 

This  conference  is  best  served  by  my  talk  if  I  list  some  of  the  things  that  have  been  learned  about 
the  spatio-temporal  properties  of  tactile  displays  that  could  be  useful  in  the  design  of  displays  for 
teleoperator  systems.  One  of  these  has  to  do  with  the  site  of  the  display,  which  is  commonly  of  con¬ 
cern  to  sensory  aids  developers  for  reasons  of  comfort  and  appearance.  In  addition  to  such  concerns, 
however,  it  has  become  clear  that  issues  of  efficiency  of  energy  usage,  related  to  the  thresholds  for 
feeling,  and  the  needed  dynamic  range  of  such  devices,  are  also  important  to  the  designer,  and  these 
parameters  vary  greatly  with  the  site  of  stimulation.  Of  equal  importance  is  spatial  acuity,  i.e.,  the 
discrimination  of  high  spatial  frequencies  on  the  skin,  which  varies  systematically  with  the  location 
of  the  stimulus  array. 

There  is  a  considerable  literature  dealing  with  such  space-time  phenomena  as  masking,  real  and 
apparent  movement  on  the  skin,  temporal  integration  of  both  energy  and  coded  information,  salta¬ 
tion,  and  lateralization.  These  will  be  discussed  in  some  detail  as  useful  boundary  conditions  for 
device  design. 

Another  class  of  phenomena  exist  that  may  appear  at  first  to  be  too  esoteric  for  applied  consid¬ 
eration,  but  which,  because  they  have  only  a  short  history  of  systematic  research,  deserve  considera¬ 
tion  for  their  potential  in  enriching  the  perceptual  field  of  the  tactile  display.  One  of  these  is  the 
interaction  of  touch  with  other  modalities,  and  a  good  example  is  found  in  studies  of  the  temperature 
sense.  Another  is  the  phenomenon  of  distal  attribution,  which  involves  the  transference  of  the  display 
properties  from  the  veridical  site  of  stimulation  to  the  object  being  simulated  by  the  display. 
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A  final  word  will  be  said  about  the  importance  of  the  learning  process  in  acquiring  a  little-used 
perceptual-motor  skill,  which  teleoperator  systems  must  demand  of  the  user  when  an  unusual  source 
such  as  touch  is  incorporated  into  the  information  stream. 
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NEURAL  MECHANISMS  OF  KINESTHESIA 


Francis  J.  Clark 
University  of  Nebraska 
Omaha,  Nebraska 


Our  primary  interest  is  in  where  the  signals  for  position  and  movement  of  the  limbs  originate  and 
how  the  information  is  encoded.  Humans  can  independently  sense  the  positions  and  movements  of 
their  limbs  in  the  absence  of  vision,  and  these  senses  are  derived  from  mechanoreceptors  located  in 
the  limbs.  Receptors  in  muscles,  skin  and  joints  are  potential  sources  of  position  and  movement  sig¬ 
nals,  but  how  (and  whether)  each  of  these  groups  contributes  to  kinesthesia  remains  uncertain. 

Recent  evidence  indicates  that  a  sense  of  static-position  derives  from  length  receptors  in  muscle, 
whereas  receptors  in  the  skin,  and  possible  though  not  likely  the  joints,  can  signal  movement  of  a 
limb  but  not  its  static  position. 

We  distinguish  this  static-position  sense  from  a  movement  sense  on  the  basis  of  whether  a  sub¬ 
ject’s  ability  to  sense  a  small  displacement  (3  to  5  deg)  of  a  joint  varies  with  the  rate  of  joint  dis¬ 
placement.  Therefore,  if  subjects  possess  an  awareness  of  the  static-position  of  a  joint,  slow  rates  of 
rotation  should  not  seriously  degrade  their  ability  to  sense  displacements.  In  the  absence  of  a  static- 
position  sense,  subjects  would  rely  on  movement  signals  that  do  depend  on  the  rate  of  rotation  and 
their  ability  to  detect  displacements  should  decrement  with  decreasing  rate.  This  is  roughly  analo¬ 
gous  to  distinguishing  a  low-pass  filter  (a  static-position  sense)  from  a  high-pass  filter )  a  movement 
sense  only)  by  their  frequency  response  characteristics. 

The  interphalangeal  (IP)  joints  of  the  fingers  (and  probably  the  toes)  appear  to  lack  a  static-posi¬ 
tion  sense.  Subjects  were  unable  to  detect  large  displacements  of  the  IP  joints  if  the  displacements 
were  slow  enough  (our  tests  used  rates  less  than  1  deg/min).  In  contrast,  small  displacements  of  the 
metacarpophalangeal  (MCP)  joint  of  the  finger  could  be  detected  at  the  slowest  rates  we  used.  This 
is  a  surprising  finding  given  that  people  seem  quite  unaware  they  have  this  kinesthetic  deficit  at  the 
fingers. 

At  present,  we  are  examining  the  consequences  of  not  having  a  static-position  sense  at  the  fingers 
apart  from  an  inability  to  detect  very  slow  displacements.  Using  a  more  demanding  test,  we  have 
found  no  substantial  difference  in  the  accuracy  of  reproducing  target  positions  passively  imposed  on 
the  MCP  joint  and  the  proximal  interphalangeal  (PEP)  joint  of  the  index  finger  (the  MCP  has  and  the 
PIP  lacks  a  static-position  sense).  Our  working  hypothesis  is  that  subjects  normally  use  movement 
sense  mechanisms  to  detect  displacements,  probably  because  they  are  faster,  but  how  (or  whether) 
subjects  manage  to  obtain  an  absolute  position  reference  with  only  a  movement  sense  available 
remains  unknown  and  is  an  issue  under  investigation. 

We  have  also  shown  that  subjects  can  reproduce  target  positions  of  a  joint  from  memory  about  as 
accurately  as  they  can  match  the  position  of  the  joint  to  its  corresponding  joint  on  the  opposite  limb. 
Reproducing  targets  from  memory  remains  remarkably  accurate  even  after  24  hours.  The  existence 
of  a  stable,  long-term  memory  for  limb  position  is  an  important  component  of  a  model  for  motor 
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control  we  are  investigating.  Irrespective  of  the  mechanism,  the  accuracy  of  this  memory  compli¬ 
cates  the  study  of  position  and  movement  senses.  For  example,  one  cannot  study  a  static-position 
sense  by  moving  a  joint  to  some  position  and  waiting  until  all  movement  signals  have  faded  away. 
With  a  memory,  one  could  not  distinguish  a  true  static-position  mechanism  from  a  mechanism  that 
derived  position  from  a  velocity  signal  and  stored  the  result  in  a  memory. 
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ANALYSIS  OF  THE  HUMAN  OPERATOR  SUBSYSTEMS 


Lynetve  A.  Jones  and  Ian  W.  Hunter 
McGill  University 
Montreal,  Quebec,  CANADA 


Except  in  low-bandwidth  systems,  knowledge  of  the  human  operator  transfer  function  is  essential 
for  high-performance  telerobotic  systems..  This  information  has  usually  been  derived  from  detailed 
analyses  of  tracking  performance,  in  which  the  human  operator  is  considered  as  a  complete  system 
rather  than  as  a  summation  of  a  number  of  subsystems,  each  of  which  influences  the  operator’s  out¬ 
put.  Limitations  in  analytic  techniques  and  insufficient  knowledge  about  the  characteristics  of  the 
human  operator  have  resulted  in  these  “black  box”  depictions  of  tracking  performance.  One  conse¬ 
quence  of  this  approach  is  that  it  has  been  impossible  to  determine  the  source  of  change  in  the 
parameters  of  the  transfer  function  under  different  conditions.  Studies  of  one  of  these  subsystems, 
the  limb  mechanics  system,  demonstrate  that  large  parameter  variations  can  occur  (e.g.,  the  elastic 
stiffness  of  a  limb  can  change  over  a  50-fold  range  for  different  levels  of  muscle  activation),  that  can 
have  a  profound  effect  on  the  stability  of  force-reflecting  telerobot  systems. 
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One  of  the  objectives  of  our  research  is  to  decompose  the  performance  of  the  human  operator 
system  (i.e.,  its  damping,  natural  frequency,  pure  delay,  static  gain)  in  terms  of  the  subsystems 
depicted  above,  in  order  to  establish  how  the  dynamics  of  each  of  these  elements  influence  the 
operator’s  responses.  For  example,  the  delays  associated  with  each  subsystem  are  different  (ranging 
from  15ms  for  contractile  dynamics  to  over  150ms  for  cognitive  processing),  and  for  some  subsys¬ 
tems  the  transformations  are  linear  and  time  invariant,  whereas  for  others  they  are  nonlinear  and  time 
varying.  Recent  advances  in  nonlinear  system  identification  make  it  possible  to  characterize  the 
dynamics  of  the  human  operator  subsystems  in  a  single  experimental  session.  Ideally  the  same 
equipment  and  analytic  procedures  should  be  used  in  all  phases  of  this  process.  On  the  basis  of  the 
results  from  such  an  analysis,  it  should  be  possible  to  predict  for  any  given  operator  the  conditions 
(e.g.,  muscle-joint  segment,  type  of  controlled  response,  mechanical  properties  of  the  human 
interface)  under  which  control  is  optimal. 
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With  this  objective  in  mind,  we  have  constructed  an  apparatus  for  studying  the  human  neuro¬ 
muscular  system  in  terms  of  its  neural  (contractile  dynamics),  mechanical  (limb  mechanics),  percep¬ 
tual  (sensory  thresholds),  and  tracking  (human  interface)  characteristics.  The  apparatus  consists  of 
two  powerful  computer-controlled  linear  actuators  instrumented  with  displacement  and  force  trans¬ 
ducers.  With  this  equipment  the  neuromuscular  function  of  the  muscles  controlling  the  elbow  can  be 
determined  by  imposing  small,  wide-bandwidth,  stochastic  length  perturbations  to  the  forearm.  In 
addition  the  mechanical,  reflex,  and  neural  excitation  to  force  dynamics  of  the  neuromuscular  system 
controlling  the  elbow  flexor/extensor  muscles  can  be  measured,  sensory  thresholds  can  be  calculated, 
and  finally  the  tracking  dynamics  of  the  human  operator  control  system  can  be  determined.  To  date, 
studies  of  these  subsystems  have  been  limited  to  single  joints  such  as  the  elbow,  and  the  effects  on 
tracking  performance  of  varying  the  response  type  and  human-interface  properties  have  been 
investigated. 

The  characteristics  of  the  human  operator  in  a  pursuit-tracking  task  change  as  a  function  of  the 
operator’s  response.  When  subjects  track  a  visually  presented  target  either  by  changing  the  position 
of  their  forearm  or  by  modulating  the  forces  generated  by  the  elbow  flexor  and  extensor  muscles, 
force  control  is  superior  to  position  control  in  terms  of  response  delay  (100  ms  less),  but  subjects  are 
more  accurate  when  position  is  the  controlled  variable.  Position  control  is,  however,  influenced  by 
the  mechanical  properties  of  the  human  interface,  and  in  particular  changes  in  manipulator  stiffness 
affect  both  response  delay  (less  at  higher  [up  to  2000  N/m]  stiffness  amplitudes)  and  tracking  accu¬ 
racy  (more  accurate  at  lower  [<  1500  N/m]  amplitudes). 
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INTERSENSORY  COORDINATION 


J.  R.  Lackner 
Brandeis  University 
Waltham,  Massachusetts 


Perception  of  body  orientation  and  configuration  is  dependent  on  multiple  sources  of  afferent  and 
efferent  information  about  the  spatial  configuration  of  the  body  and  its  relationship  to  the  support 
and  force  characteristics  of  the  surroundings.  Such  information  allows  us  under  normal  circum¬ 
stances  to  preserve  an  accurate  and  stable  distinction  between  thos'*  changes  in  receptor  activity  con¬ 
tingent  on  self-motion  and  those  contingent  on  motion  of  or  within  the  environment.  Maintenance  of 
this  distinction  is  essential  for  the  ongoing  control  of  normal  body  movement  and  posture. 

Because  of  this  active  interrelating  of  sensory  and  motor  cues  whenever  there  is  an  error  in  the 
registration  of  ongoing  body  orientation,  auditory  and  visual  mislocalizations  of  related  magnitude 
and  time  course  can  occur.  As  a  consequence,  depending  on  the  perceived  spatial  representation  of 
the  body,  identical  patterns  of  arrival  time  and  intensity  cues  at  the  ears  (and  identical  retinal  and 
oculomotor  signals)  can  give  rise  to  the  perception  of  sounds  (and  visual  objects)  in  widely  disparate 
spatial  positions  in  relation  to  the  head  and  to  the  external  environment.  The  interaction  between  sen¬ 
sory  localization  and  the  representation  of  body  orientation  is  actually  bidirectional.  For  example,  a 
rotating  sound  field  can  elicit  illusory  self-rotation  and  compensatory  eye  movements  in  a  stationary 
listener. 

The  conscious  appreciation  of  limb  position  is  influenced  by  muscle  afferent  signals.  When  the 
body  or  tendon  of  a  muscle  is  mechanically  vibrated,  for  example  at  120  Hz,  the  muscle  reflexly 
contracts.  This  contraction  is  known  as  a  tonic  vibration  reflex  (TVR).  If  the  motion  of  a  limb  mov¬ 
ing  under  the  action  of  a  TVR  is  resisted,  illusory  motion  of  the  (unseen)  stationary  limb  will  be 
experienced.  For  example,  if  the  biceps  brachii  muscle  is  vibrated,  the  restrained  forearm  will  be  felt 
to  move  into  extension.  A  target  light  attached  to  the  stationary  index  finger  of  the  hand,  in  an 
otherwise  dark  room,  will  be  seen  to  move  in  keeping  with  the  apparent  motion  of  the  arm  even 
though  the  eyes  maintain  steady  fixation.  If  the  subject  is  allowed  to  see  his  index  finger  in  relation 
to  the  surroundings,  but  not  the  rest  of  his  forearm,  he  will  see  his  finger  in  one  spatial  location  and 
feel  the  rest  of  his  arm  to  be  in  another  spatially  distinct  location. 

During  body  movement  in  a  high  force  field,  the  relationship  between  muscle  activity  and  mus¬ 
cle  spindle  feedback  is  altered,  much  as  in  the  case  of  the  vibration  illusions  just  described.  If  an 
individual  does  deep  knee  bends  during  exposure  to  twice  earth  gravity,  then  he  will  misperceive  the 
extent  of  his  own  voluntary  motion  and  simultaneously  experience  visual  motion  of  his  surroundings 
by  the  amount  of  that  misperception.  This  demonstrates  that  there  is  a  dynamic  relationship  between 
intended  movements  and  the  actual  movements  executed,  and  the  perception  of  those  movements. 

During  normal  terrestrial  locomotion,  deformation  of  the  optic  projection  of  the  visual  world,  the 
intensity  and  timing  of  efferent  signals  and  of  muscle  afferent  signals,  and  the  distribution  of  contact 
forces  of  support  on  the  body  co-vary  and  are  mutually  corroborative.  If  the  relationship  between 
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voluntary  stepping  movements,  body  d  splacement,  and  visual  flow  are  artificially  altered,  it  is  pos¬ 
sible  to  distort  the  perception  of  voluntary  actions  so  that  an  individual  voluntarily  stepping  forward 
can  perceive  himself  to  be  voluntarily  walking  backwards,  or  to  be  stepping  more  slowly  or  more 
rapidly  than  he  actually  is.  Common  to  all  the  remappings  elicited  are  combinations  of  perceived 
self-motion,  perceived  limb  activity,  seen  motion  of  the  visual  world,  and  felt  motion  of  the  ground 
that  are  dimensionally  consisted  (even  if  not  physically  probable,  such  as  changes  in  apparent  leg 
length)  with  minimal  departures  from  terrestrial  constraints. 
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WATCH  WHAT  YOU  SAY,  YOUR  COMPUTER  MIGHT  BE  LISTENING; 
A  REVIEW  OF  AUTOMATED  SPEECH  RECOGNITION 


Stephen  De  Gennaro 
IBM  Thomas  J.  Watson  Research  Center 
Yorktown  Heights,  New  York 


Spoken  language  is  the  most  convenient  and  natural  means  by  which  people  interact  with  each 
other  and  is,  therefore,  a  promising  candidate  for  human-machine  interactions.  Speech  also  offers  an 
additional  channel  for  hands-busy  applications,  complementing  the  use  of  motor  output  channels  for 
control. 

Current  speech  recognition  systems  vary  considerably  across  a  number  of  important  characteris¬ 
tics,  including  vocabulary  size,  speaking  mode,  training  requirements  for  new  speakers,  robustness  to 
acoustic  environments,  and  accuracy.  Algorithmically,  these  systems  range  from  rule-based  tech¬ 
niques  through  more  probabilistic  or  self-learning  approaches  such  as  Hidden  Markov  Modeling  and 
Neural  Networks. 

This  tutorial  begins  with  a  brief  summary  of  the  relevant  features  of  current  speech  recognition 
systems  and  the  strengths  and  weaknesses  of  the  various  algorithmic  approaches. 

Issues  critical  to  the  successful  application  of  speech  recognition  in  human-machine  interactions 
will  be  discussed  in  more  detail.  These  issues  include:  “raw”  recognition  vs.  speech  understanding, 
grammar-based,  coi.. strained  dialog  vs.,  natural-language,  free  discourse  and  the  imposition  of 
appropriate  constraints  to  make  the  task  technologically  feasible,  with  acceptable  accuracy.  The 
potential  delay  in  recognition,  due  to  algorithmic  features  or  to  interactive  verification  of  recognized 
words  for  error  correction,  must  also  be  considered.  The  need  for  verification  is  determined  by  both 
the  expected  eixor  rate  and  the  cost  of  each  error.  The  cost  function  could  be  dynamic  if  errors  will 
have  significantly  different  impact  given  the  state  of  the  system. 

.As  in  any  human-machine  interface,  not  all  of  the  constraints  are  due  to  the  “-machine”  half  of 
the  interaction;  the  "human-”  partner  also  imposes  a  share  of  burdens.  For  example,  people  will  not 
always  say  exactly  what  is  expected  of  them,  even  when  carefully  prompted  with  a  small  set  of  pos¬ 
sible  choices.  Stress,  attention,  and  general  physical  state  can  have  strong  effects  on  system  perfor¬ 
mance. 


For  applications  that  are  not  currently  speech-driven,  it  might  also  be  necessaiy  to  reforiTialate 
the  basic  human-machine  interface.  For  example,  moving  a  cursor  over  a  document  by  voice  in  the 
most  straightforward  way  (up-up-up-left-left-  ...^  is  not  as  efficient  (and  probably  not  as  acceptable) 
as  moving  the  cursor  by  keys  or  mouse.  Recasting  the  basic  problem  as  “go  to  the  word  HELLO  is 
the  second  paragraph”  provides  a  better  match  between  the  task  and  spee  h  control.  Navigating  a 
robot  by  voice  instead  of  by  joystick  might  require  .similar  reformulation.  Of  course,  while  “go  to  the 
word  ..."  is  more  natural,  it  imposes  new  requirements  to  understand  the  command  and  map  it  into 
the  appropriate  action. 
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Given  these  characteristics,  it  is  not  yet  reasonable  to  expect  that  an  off-the-shelf  speech  recog¬ 
nizer  can  be  used  as  a  transparent  replacement  for  a  keyboard  or  joystick  as  input  to  any  arbitrary 
application.  The  highest  performance  levels  will  be  achieved  when  the  application  and  recognition 
processes  are  more  tightly  coupled. 

Using  existing  speech  recognition  technology,  it  is  certainly  possible  to  build  successful  virtual 
environments.  This  is  particularly  true  for  systems  that  mimic  environments  in  which  speech  is 
already  the  primary  communication  mode,  and  where  the  control  “protocols”  have  been  well 
defined.  For  example.  Air  Traffic  Control  training  consoles  have  been  demonstrated  using  continu¬ 
ous  speech,  but  the  task  is  constrained  to  deal  only  with  objects  (i.e.,  specific  flights)  on  the  trainee’s 
-onsolc  to  provide  reasonable  accuracy  and  response  delay.  A  general  problem  with  coupling,  how¬ 
ever.  is  that  it  leaves  the  system  vulnerable  to  unanticipated  scenarios,  with  concomitant  risk  that  the 
system  will  not  respond  gracefully  to  “exceptional”  interactions. 

This  tutorial  will  conclude  with  some  pointers  to  conferences  and  publications  that  reflect  the 
current  state  of  commercially  available  speech  recognizers,  and  the  state  of  research  in  this  field. 
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EYE  MOVEMENT  CONTROL  AND  TOP-DOWN  SCANPATH  VISION  AS  A 
DESIGN  METAPHOR  FOR  ROBOTIC  VISION  AND  CONTROL 

Lawrence  W.  Stark 
University  of  California 
Berkeley,  California 


I.  Eye  Movement  Control 

1 .  Kinetics:  Time  Optimal  Control  of  Saccades 

Eye  movements  are  elegant  biological  examples  for  application  of  time-optimal  engineering 
control  theory.  The  muscles  that  drive  eyeball  rotations  are  very  long  and  strong  and  fast.  The 
eyeball  has  low  rotational  inertia  so  that  the  internal  viscosity  of  active  muscle  dominates  plant 
dynamics;  thus  we  may  think  of  the  system  as  a  visco-elastic  load.  What  controller  signal  shapes 
would  produce  a  time-optimal  movement?  First,  reciprocal  innervation,  proposed  by  Descartes 
(#149)  and  experimentally  found  by  Sherrington,  should  be  observed  (Figure  lA).  Next,  a  set  of 
multipulse-step  control  signals  first  described  by  Clark  and  Stark  (#137)  can  be  justified  on  theoreti¬ 
cal  grounds  by  Bellman-Pontriagin  bang-bang  theory. 

2.  Dual  Mode  Control:  Sampled  Data  Operator 

Another  beautiful  example  is  the  sampled  data  control  model  of  Young  and  Stark  (#27)  that 
was  experimentally  verified  by  study  of  transient  responses  in  normal  subjects.  A  further  exciting 
demonstration  was  found  in  the  explanation  of  “macro-saccadic  oscillation”  found  in  patients  with 
cerebellar  disease  that  effectively  removed  the  continual  adaptive  gain  adjustment  and  left  the  patient 
with  high  gain  instability  oscillation  (#156).  (Figure  IB). 

II.  Higher  Level  EM  Control  and  Scanpath  Theory 

3.  Active  Looking  and  the  Kantian  Theory  of  Perception 

A  more  metaphoric  theory,  the  Scanpath  Theory  has  been  developed  by  Noton  and  Stark 
(#1 10)  and  Stark  and  Ellis  (#211)  to  account  for  the  repetitive  sequence  of  saccades  that  enable  the 
eye  and  its  fovea  centralis  to  traverse  over  important  sub-features  of  a  picture  or  scene.  This  has  been 
taken  to  mean  than  an  internal  cognitive  model  drives  active  looking  in  a  top-down  manner.  The  evi¬ 
dence  to  support  this  theory  has  to  do  with  statistical  proof  that  there  is  indeed  repetition  of  scanpath 
sequences.  Further  evidence  comes  from  studies  of  ambiguous  figures  wherein  the  same  subject 
looking  at  the  same  picture  switches  both  mental  states  of  perception  and  patterns  of  repetitive  sac¬ 
cadic  sequences.  Most  recently,  Brandt  and  Stark  (in  preparation)  have  used  visualization  tasks.  A 
subject  recalling  a  previously  seen  diagram  demonstrates  scanpath  movements  similar  to  those 
looking  at  the  earlier  viewed  diagram;  here  only  the  internal  model  can  be  responsible  for  the  scan¬ 
path  sequence  and,  indeed,  string  editing  distances  support  this  finding  quantitatively.  (Figure  2A). 
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The  philosopher,  Kant,  reasoned  that  internal  representations  (Platonic  ideals)  controlled  active 
perception  in  a  top-down  procedure  that  could  then  organize  sensation  (without  time  and  without 
space)  received  from  the  chaotic  work  of  apptearances.  In  this  view,  the  virtual  model  within  a  human 
brain  is  more  real  than  the  external  world  of  the  naive  realistic!  (Figure  2B). 


Acknowledgment  to:  NASA  Ames  Research  Center  Cooperative  Agreement  NCC-286,  Dr.  Stephen 
Ellis,  Technical  Monitor,  and  to  California  Institute  of  Technology,  Jet  Propulsion  Laboratories, 

Dr.  Anatal  Bejczy,  JPL  Technical  Manager,  for  their  support. 
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III.  Top-Down  Robotic  Vision 


4.  Models  of  Robots  and  ROIs 

For  our  use  of  visual  feedback,  for  use  by  autonomous  algorithms,  for  use  by  manual  operators, 
and  for  use  by  supervisory  controllers.  We  have  taken  this  top-down  scheme  from  human  vision  and 
introduced  it  as  a  basic  design  principle  in  our  Telerobotic  Laboratory.  Instead  of  searching  for 
important  features  of  the  TRWE  (telerobotic  working  environment)  with  bottom-up  image  process¬ 
ing  algorithms,  we  instead  use  a  top-down  approach.  We  already  know  where  a  particular  robotic 
link  is  located  in  3D  space,  we  know  where  the  cameras  are  pointed,  we  know  what  important 

parameters  we  need  to  measure.  Thus,  we  use  our  knowledge  in  the  form  of  forward-loop  control 
model  not  only  to  control  the  robots  and  me  cameras,  but  also  to  direct  the  image  processing  (#312 
and  A304).  We  construct  ROIs  (regions-of-interest)  for  each  camera  frame  so  that  the  image 
processing  can  be  done  lapidly  and  robustly  (Figure  3 A). 

Within  the  ROIs,  local  thresholding  and  centroid  calculations  enable  us  to  compute  a  feedback 
niodel,  indicated  by  crosses  that  represent  the  visual  measurement  of  the  location  of  key  on-the- 
scene-visual-enhancements.  This,  of  course,  is  done  dynamically  so  that  rapid  feedback  permits 
closed-loop  control  of  the  robotic  movement,  with  the  human  operator  acting  as  a  supervisory  con¬ 
troller  only  (Figure  3B). 

IV.  Over- All  Robotic  Control  Scheme 

5.  Model  of  RWE 

The  overall  model  (Figure  4A)  permits  one  to  appreciate  the  conununication  functions  of  our 
top-down  image  processing  scheme.  Communication  is  restricted  to  parameters  of  the  visual  models, 
an  enormous  bandwidth  reduction  compared  to  sending  video  pictures  of  a  number  of  views  of  the 
TRWE.  The  supervisory  controller  looks  not  at  the  camera  view  of  the  scene,  but  rather  at  a  model  of 
the  TRWE!  Is  this  going  to  be  satisfactory?  (#T41l,  #313). 

6.  Evaluation  of  Displays  and  HMD 

Several  evaluation  studies  of  visual  display  requirements  have  been  carried  out  in  our  labora¬ 
tory  (#301,  #A313).  Dr.  Won  Soo  Kim  and  colleagues  have  established  that,  although  for  some  pur¬ 
poses  enhanced  2D  displays  enable  the  operator  to  perform  satisfactorily,  virtual  3D  displays  are 
robust  to  various  deficiencies  and  enable  performance  at  least  as  good  as  the  best  2D  displays.  Also, 
models  alone  or  models  superimposed  onto  video  pictures  are  better  than  video  pictures  for  short¬ 
term  robotic  control  tasks  (Figure  4C,  #A291).  In  another  study.  Dr.  Greg  Tharp  and  colleagues 
demonstrated  that  5  to  15  Hz  update  rates  for  a  stereo  display  were  fast  enough  to  permit  the  subject 
to  slew  his  head  freely  without  suffering  a  penalty  in  performance  (delays  were  less  than  1  frame 
interval).  (Figure  4B). 
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V.  Summary 


Telerobotics  with  virtual  environments  is  an  exciting  and  worthwhile  field  of  research.  It  offers 
challenges  to  engineering  design  of  displays,  of  control,  and  of  communication  schemes.  It  relates  to 
human  performance  since  the  human  operator  is  a  component  either  as  a  direct  manual  controller  or 
as  a  supervisory  conf'-Uer  h  wi|!  teach  us  about  human  capacities  and  we  can  use  knowledge 
of  human  plans  and  schemas  to  produce  initial  designs.  We  are  also  learning  more  about  ourselves. 
Did  we  thing  about  “presence”  before  we  considered  telepresence? 
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Figure  lA.  Descartes  model. 

#149  Kenneth  J.  Ciuffreda  and  Lawrence  Stark,  “Descartes’  Law  of  Reciprocal  Innervation,”  American  Journal  of 
Optometry  and  Physiological  Optics  52:  663-673  (1975) 
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Figure  IB.  SD  clinical  simulation. 

#27  Laurance  R.  Young  and  Lawrence  Stark,  “Variable  Feedback  Experiments  Testing  a  Sampled  Data  Model  for  Eye 
Tracking  Movements.”  IEEE  Transactions  on  Human  Factors  in  Electronics  HFE-4:  3S-51  (1963) 

#156  John  B.  Selhourst,  Lawrence  Stark,  Alfred  L.  Ochs  and  William  F.  Hoyt,  “Disorders  in  Cerebellar  Ocular  Motor 
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SCANPATH  THEORY 


Figure  2A.  Scanpath  theory. 

#110  David  Noton  and  Lawrence  Stark,  “Eye  Movements  and  Visual  Perception.”  Scientific  American  224:  334-43 
(1971). 

#211  Lawrence  Stark  and  Stephen  Ellis,  “Scanpaths  Revisited:  Cognitive  Models  Direct  Active  Looking.”  In:  Eye 
Movements,  Cognition  and  Visual  Perception,  ed.  Fisher,  Monty  and  Senders,  New  Jersey:  Eribaum  Press,  193-226 
(1981). 
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Figure  2B. 
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Figure  3A  and  3B. 
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Figure  4A. 
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Figure  4C. 
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TELEOPERATORS 


TELEROBOTIC  REMOTE  PRESENCE:  ACfflEVEMENTS  AND 

CHALLENGES 


Walter  A.  Aviles 
Naval  Ocean  Systems  Center 
Kailua,  Hawaii 


The  Marine  Sciences  and  Technology  Department  of  the  Naval  Ocean  Systems  Center  has  as  one 
of  its  particular  thrusts  the  development  of  telerobotic  work  systems  which  blend  the  capabilities  of  a 
human  with  those  afforded  by  automation  technologies.  The  goal  of  these  systems  is  to  give  a  human 
operator  the  subjective  experience  of  being  at  a  remote  work  site  when  he  is  in  fact  at  a  location  well 
removed  from  the  (potentially  hazardous)  work  area.  Systems  of  this  type  are  said  to  exhibit  “tele¬ 
presence”.  Telepresence  is  accomplished  through  the  use  of  spatially  correspondent  controls  and 
high-fidelity  sensory  feedback  systems  which  utilize  the  human  operator’s  cognitive  and  sensory 
capabilities  in  a  manner  that  is  natural  to  him.  A  major  advantage  of  the  telepresence  approach  is  that 
the  human  operator  does  not  have  to  concentrate  on  how  to  use  unfamiliar  or  unnatural  displays  and 
controls  but  instead  uses  his  own  cognitive  and  manipulative  abilities  to  perform  the  task.  As  a  con¬ 
sequence,  the  telepresence  approach  minimizes  the  time  required  to  perform  work  as  well  as  mini¬ 
mizing  the  amount  of  training  required  to  use  sophisticated  undersea  work  systems. 

Although  the  goals  of  a  telerobotic  human-machine  interface  exhibiting  telepresence  are  intu¬ 
itively  straightforward  to  grasp,  the  quantification  of  terms  such  as  “correspondent  controls”  and 
“high-fidelity  sensory  feedback”  in  a  manner  useful  for  development  remains  a  major  challenge  to 
the  field.  In  addition,  the  proper  blending  of  automated  assistance,  remote  work  site  sensory  feed¬ 
back,  and  artificially  generated  sensory  feedback,  is  both  a  fascinating  and  challenging  endeavor. 
This  talk  will  attempt  to  outline  the  progress  in  human-machine  interfaces  from  the  perspective  of 
telerobotic  remote  presence  and  to  suggest  avenues  for  future  research. 
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ISSUES  IN  THE  DESIGN  OF  HIGH  DEXTERITY,  FORCE  REFLECTIVE 

TELEOPERATORS 


S.  C.  Jacobsen,  E.  K.  Iversen,  C.  C.  Davis,  K.  B.  Biggers,  and  D.  K.  Backman 

University  of  Utah 
Salt  Lake  City,  Utah 


The  Center  for  Engineering  Design  at  the  University  of  Utah  is  developing  an  anthropomorphic, 
hydraulically  actuated,  teleoperated  arm.  The  system  includes  a  sixteen  degree-of-freedom  slave 
manipulator  controlled  by  a  kinematically  identical,  sixteen  degree-of-freedom  force-reflective, 
exoskeletal  master.  The  project  has  focused  on  four  areas:  (a)  formulating  a  realistic  t  of  design 
objectives  which  balance,  against  technical  realities,  the  desire  for  performance,  reliability  and  econ¬ 
omy,  (b)  understanding  control  issues,  (c)  designing  and  fabricating  new  subsystems  (presently 
unavailable)  necessary  for  the  construction  of  a  successful  machine  and  (d)  integratin,  >1  Ssystems, 
through  a  series  of  prototype  stages,  into  an  operational  teleoperation  system. 

Generating  a  comprehensive  set  of  design  objectives  required  the  consideration  of  fundamental 
questions  such  as:  (1)  what  should  be  the  kinematic  configuration  of  master  and  slave,  hand  and  arm, 
(2)  how  many  degrees  of  freedom  will  be  necessary  for  the  level  of  dexterity  desired,  (3)  should  the 
master  be  exoskeletal,  or  some  other  configuration,  (4)  what  is  the  importance  of  force  reflection  and 
how  can  its  fidelity  be  improved  beyond  current  systems,  (5)  what  are  the  realistic  actuation  alterna¬ 
tives  considering  active  and  passive  performance  requirements,  (6)  what  performance  is  required  of 
sensor  systems,  (7)  when  necessary,  can  tendons  be  successfully  used  to  remotize  actuators. 

Our  work  in  the  control  area  focused  on:  (1)  developing  stable  position  and  torque  feedback 
approaches  which  result  in  desirable  inter-system  impedance  (frequency  dependent  gain,  stiffness 
and  damping),  (2)  generating  acceptable  joint-torque  compensation  which  removes  undesirable 
effects  of  gravity,  mass  and  viscous  drag,  (3)  increasing  system  bandwidth  to  improve  operator 
awareness  of  loads  imposed  on  the  slave. 

Extensive  efforts  were  expended  in  the  design  and  fabrication  of  presently  unavailable  subsys¬ 
tems  such  as:  (1)  rotary  and  linear  hydraulic  actuators  along  with  required  servo  valves,  (2)  joint- 
torque  sensors  capable  of  remote-axis-sensing,  high  stiffness,  low  stiction  and  minimal  sensing  of 
loads  on  other  axes,  (3)  position  sensors  with  acceptable  accuracy,  (4)  computation  systems  which 
include  digital  and  analog  electronics  for  implementation  of  servo,  intermediate  and  higher  level 
control  functions,  (5)  systems  for  long  range  communication  between  the  master  and  slave  systems. 
Designing  subsystems  such  that  they  “package”  into  the  available  shaped  volumes  as  well  as  satisfy¬ 
ing  other  constraints  was  a  problem  of  substantial  magnitude.  Packaging  remains  an  underestimated 
problem  which  in  fact  dominates  many  design  decisions. 

Finally,  work  is  proceeding  towards  integrating  subsystems  into  a  working  machine  which 
actually  achieves  its  objectives  with  respect  to  performance,  reliability  and  economy.  The  system 
must:  ( 1 )  be  capable  of  operating  in  sea  environments  which  includes  corrosion  and  substantial 
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pressure,  (2)  survive  substantial  mechanical  abuse  and  misuse,  (3)  include  mechanical  machine- 
operator  interfaces  which  injure  comfortable,  nonfatiguing  operation  over  long  periods  of  time. 

The  presentation  will  address  the  i:sues  mentioned  above  as  well  as  include  a  video  demonstra¬ 
tion  of  the  system  executing  selected  dextrous  tasks. 
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TELE-EXISTENCE  AND/OR  CYBERNETIC  INTERFACE  STUDIES 

IN  JAPAN 


Susumu  Tachi 

Mechanical  Engineering  Laboratory,  MITI 
Tsukuba  Science  City,  P^araki,  JAPAN 


Tele-existence  aims  at  a  natural  and  efficient  remote  control  of  robots  by  providing  the  operator 
with  a  real  time  sensation  of  presence.  It  is  an  advanced  type  cf  teleoperation  system  which  enables  a 
human  operator  at  the  controls  to  perform  remote  manipulation  tasks  dexterously  with  the  feeling 
•^^hat  he  or  she  exists  in  one  of  the  remote  anthropomorphic  robots  in  the  remote  environment,  e.g.,  in 
a  hosiile  environment  such  as  those  of  nuclear  radiation,  high  temperature,  and  deep  space. 

Systematic  research  for  the  development  of  tele-existence  has  been  conducted  at  MEL  by  feeding 
back  rich  sensory  information  which  the  remote  robot  has  acquired  to  provide  the  operator  with  a 
real-time  sensation  of  presence. 

With  the  advent  of  science  and  technology,  it  has  become  possible  to  envision  teleoperation  with 
a  sensation  of  presence.  The  concept  of  projecting  ourselves  by  using  robots,  computers  and  cyber¬ 
netic  human  interfaces  is  called  Tele-Existence,  and  a  lot  of  effort  has  been  made  for  the  realization 
of  this  concept.  This  concept  also  provides  an  extension  of  human  sensing  and  muscular  capabilities. 

Tne  tele-existence  system  consists  of  intelligent  mobile  robots,  their  supervisory  subsystem,  a 
remote-presence  subsystem  and  a  sensory  augmentation  subsystem,  which  allows  an  operator  to  use 
the  robot’s  ultrasonic,  infrared  and  other,  otherwise  invisible,  sensory  information  with  the  computer 
graphics-generated  pseudo-realisitic  sensation  of  presence.  In  the  remote-oresence  subsystem,  real¬ 
istic  visual,  auditory,  tactile,  kinesthetic  and  vibratory  displays  will  be  realized  (ref.  1). 

Using  this  system,  a  human  operator  can  be  in  a  safe  and  comfortable  environment  and,  at  the 
same  time,  be  present  or  exist  at  other  environments  where  the  robots  are  working.  He  or  she  will 
monitor  the  work  through  the  robot’s  sensors,  and,  if  necessary,  conduct  the  task  on  behalf  of  the 
robot  as  if  he  v.  as  doing  the  work  himself  (ideally)  or  as  if  he  was  working  inside  the  robot 
(practically). 

The  principle  of  the  tele-existence  sensory  display  has  been  proposed.  Its  design  procedure  has 
been  explicitly  defiiicd.  Experimental  visual  display  hardware  has  been  built,  and  the  feasibility  of 
the  visual  display  with  the  sensation  of  presence  has  been  demonstrated  by  psychophysical  experi¬ 
ments  using  the  test  hardware  (refs.  1,2). 

A  method  has  also  been  proposed  co  develop  a  mobile  tele-existence  system,  which  can  he 
remotely  driven  with  the  auditory  and  visual  sensation  of  presence  (ref.  3).  A  prototype  mobile  tele¬ 
vehicle  system  ha.s  been  constructed  and  tne  feasibility  of  the  method  has  been  evaluated. 
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In  order  to  study  the  use  of  the  tele-existence  system  in  the  artificially  constructed  environment, 
the  visual  tele-existence  simulator  has  been  designed,  a  pseudo-real-time  binocular  solid  model  robot 
simulator  has  been  made,  and  its  feasibility  has  been  experimentally  evaluated  (ref.  4). 

An  anthropomorphic  robot  mechanism  with  an  arm  having  seven  degrees  of  freedom  has  been 
designed  and  developed  as  a  slave  robot  for  feasibility  experiments  of  teleoperation  using  the  tele¬ 
existence  method.  An  impedance  controlled  active  display  mechanism  and  a  head  mounted  display 
have  also  been  designed  and  developed  as  the  display  sub-system  for  the  master.  The  robot’s  struc¬ 
tural  dimensions  are  set  very  close  to  those  of  humans,  and  it  is  designed  to  mimic  the  movement  of 
humans  (ref.  5).  By  using  this  tele-existence  master  slave  system,  the  feasibility  of  the  tele-existence 
master  slave  system  has  been  demonstrated. 
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TELEROBOTICS  IN  THE  DEEP  OCEAN 


Hagen  Schempf  and  Dana  R.  Yoerger 
Woods  Hole  Oceanographic  Institution 
Woods  Hole,  Massachusetts 


Remotely  operated  vehicles  can  be  made  more  useful  for  a  wide  range  of  survey  tasks  through 
the  use  of  precision  navigation  and  automatic  control  integrated  under  a  proper  supervisory  control 
framework.  Tasks  ranging  from  scientific  survey  to  ship  hull  inspection  can  be  done  more  produc¬ 
tively  and  produce  a  more  useful  result  if  the  motions  of  the  ROV  are  tightly  controlled.  A  key  ben¬ 
efit  of  automatic  control  is  the  ability  to  repeat  a  track  at  a  later  time  to  study  dynamic  processes 
seafloor. 

This  paper  presents  experimental  results  of  the  control  system  for  the  JASON  ROV  that  has  been 
designed  for  precision  survey  and  other  automated  applications.  The  JASON  control  system  empha¬ 
sizes  a  form  of  supervisory  control  where  the  human  pilot  and  the  automatic  system  share  the  control 
task.  Results  presented  included  hovering,  automatic  track  following,  and  several  interactive  modes. 

JASON  is  equipped  with  an  underwater  manipulator  that  can  exhibit  a  wide  range  of  compliance 
through  a  combination  of  mechanical  design  and  software  control  is  described  and  its  performance 
characterized.  The  major  goal  of  the  design  was  to  produce  a  manipulator  that  can  control  the  inter¬ 
action  forces  with  the  work  task  and  can  work  reliably  in  the  hostile  deep-ocean  environment.  The 
manipulator’s  performance  has  been  characterized  in  the  lab  and  its  overall  operational  utility  has 
been  confirmed  during  an  archaeological  excavation  at  700  meters  depth  in  the  Mediterranean. 
Results  from  the  lab  tests  and  sea  trials  will  be  presented. 
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MODELS  OF  REMOTE  MANIPULATION  IN  SPACE 


Blake  Hannaford 
University  of  Wasnington 
Seattle,  Washington 


Astronauts  operating  in  space  can  hardly  be  called  autonomous.  Their  progress  is  constantly 
being  monitored  and  supported  by  the  ground  team.  The  same  procedures  will  apply  to  robotic  sys¬ 
tems  in  space  -  no  matter  how  intelligent.  This  is  also  true  of  other  important  manipulation  tasks.  For 
example,  a  surgeon  must  perform  an  operation  within  a  system  of  supervision  and  documented  pro¬ 
cedures  which  ensure  quality  care  and  accountability.  What  these  two  domains  have  in  common  is  a 
high  value  environment  in  which  a  highly  specialized  manipulation  is  performed  a  small  number  of 
times  (in  contrast  to  factory  automation).  Thus  robots  involved  in  high  value  manipulation  must  be 
effectively  coupled  to  a  human  operator  either  at  the  work-site  or  remotely  connected  via  communi¬ 
cation  links. 

Hundreds  of  hours  of  man  in  the  loop  experiments  have  been  performed  to  quantify  the  perfor¬ 
mance  of  telemanipulation  systems  with  different  levels  of  capability. 

In  order  to  make  use  of  experimental  performance  evaluation  data,  models  must  be  developed. 
Powerful  models  of  remote  manipulation  by  humans  can  be  used  to  predict  manipulation  perfor¬ 
mance  in  several  ways.  First,  they  will  allow  prediction  of  performance  in  future  systems  based  on 
today’s  laboratory  systems.  In  this  paradigm,  the  models  are  developed  from  experimental  data,  and 
then  used  to  predict  performance  in  slightly  different  situations.  Second,  accurate  telemanipulation 
models  will  allow  design  of  manipulation  systems  which  extend  manipulation  capability  beyond  its 
current  bounds. 

The  barriers  which  are  currently  being  extended  by  state  of  the  art  remote  manipulation  systems 
include  distance  and  scale.  Distance  barriers  are  most  relevant  to  space  telemanipulation  although  the 
large  manipulators  made  feasible  by  the  lack  of  gravity  involve  scale  factors  of  about  10.  Micro¬ 
teleoperators  are  now  being  developed  which  couple  the  operator  to  micro-worlds.  For  example,  a 
force  reflection  system  has  recently  been  developed  by  IBM  which  couples  the  operator  to  the  tip  of 
a  scanning  tunneling  microscope:  a  scale  factor  of  about  10“^! 

The  famous  time  delay  problem  inherent  in  overcoming  the  distance  bander  is  yielding  to  new 
methods  in  which  control  is  shared  between  the  manipulator  and  the  human  operator  in  ways  which 
allow  energetic  interaction  between  manipulator  and  environment. 

Models  developed  by  several  laboratories  to  explain  the  dynamics  of  kinesthetic  force  reflection 
will  be  briefly  reviewed  and  applied  to  the  experimental  data.  Two-port  network  models  have  been 
successful  at  improving  the  understanding  and  performance  evaluation  of  teleoperators  with  kines¬ 
thetic  force  feedback  in  which  the  human  operator,  master  manipulator,  slave  manipulator,  and  envi¬ 
ronment  are  coupled  together  through  servo  control  loops.  Such  systems  can  create  a  sense  that  the 
opei  ator  directly  manipulates  the  environment. 
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A  relatively  new  model  of  task  performance  based  on  Hidden  Markov  Models,  is  able  to  analyze 
progression  of  multi-stage  compound  tasks.  The  HMM  has  several  properties  that  make  it  attractive 
for  use  as  a  near-real-time  execution  monitor  and  post  hoc  “explainer”  in  practical  applications. 
Among  them  are: 

o  Explicit  Representation  of  Uncertainty 

o  Support  for  “knowledge  based”  operation  (the  HMM  can  be  constructed  heuristically  by  an 
expert). 

o  Closed  form,  thoroughly  developed,  efficient  algorithms  for  state  estimation  (Viterbi)  and  model 
identification  (Baum-Welch). 
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TELEOPER/ITING  THE  SPACE  STATIONS 


Marvin  L.  Minsky 

Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts 

(Talk  not  presented.) 
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SOME  CRITERIA  FOR  TELEOPERATORS  AND  VIRTUAL 
ENVIRONMENTS  FROM  EXPERIENCES  WITH 
VEHICLE/OPERATOR  SIMULATION 


Henry  R.  Jex 
Systems  Technology,  Inc. 
Hawthorne,  California 


Over  30  years  of  experience  in  the  simulation,  modeling  and  analysis  of  man-machine  control 
systems  has  resulted  in  a  variety  of  rules-of-thumb  and  some  quantitative  criteria  for  assuring  valid 
simulation  of  “real-world”  situations  and  likely  users.  It  is  hoped  that  these  criteria,  which  apply 
directly  to  teleoperations  (i.e.,  no  communication  delays;  operator-in-the-loop;  no  “smart-robot”  del¬ 
egation),  will  be  useful  for  designing  better  telerobots  and  in  simulating  telerobotic  controls  and  vir¬ 
tual  environments. 

First,  some  definitions  are  given:  “Valid  simulation”  is  defined  as  that  producing  the  same  opera¬ 
tor  behavior,  performance  and  mental  workload  as  the  real-world  case.  “Delay  artifact”  is  the 
(measured)  sum  of  all  throughput  delays  from:  sensing,  computing,  pipeline  and  display  generation. 
“Kinetosis”  is  the  vertigo  and  malaise  from  conflicts  among  visual,  vestibular  and  proprioceptive 
motion  cues.  “Mental  workload”  is  the  operator’s  evaluation  of  his  attentional-load  margin  while 
performing  adequately  in  a  specified  task  context. 

A  review  is  given  of  a  wide  range  of  simulations  in  which  operator  steering  control  of  a  vehicle 
is  involved  and  the:  dominant-cues,  closed-loop  bandwidth,  measured  operator  effective  time-delay, 
and  ratio  of  bandwidth-to-inverse  delay,  are  summarized.  For  well  designed  and  low  mental  work¬ 
load  conditions,  this  ratio  is  well  under  1 .0,  as  control  theory  would  dictate.  However,  for  important 
human  tasks  like  fast  locomotion,  the  ratio  approaches  1.0  (limited  by  the  inverse  delay  limit  of 
about  8  rad/sec),  and  the  mental  workload  becomes  critical.  These  imply  requirements  that  the  eye¬ 
ball  stabilization  and  pursuit  subsystems  have  higher  bandwidth.  More  importantly,  it  dictates  that 
any  simulation  delay  artifact  must  be  less  than  about  1/4  of  the  effective  operator  delay  in  a  given 
task,  i.e.,  less  than  about  .04  seconds  delay  artifact  for  locomotion  simulation. 

A  correlation  of  kinetosis  (simulator-sickness)  with  dynamic  scene  field-of-view  (FOV)  is 
shown.  It  has  been  found  that,  to  avoid  kineto.sis,  the  FOV  should  span  less  lateral  view  than  a  curve 
of  about  1 . 10  Gy  would  cover  at  real-world  speeds  (typically,  less  than  60  degrees).  Other  require¬ 
ments  include:  framing  of  scene  “streamers”  and  avoidance  of  perceivable  image  jitter  or  “strobing.” 

The  visual  resolution  of  a  virtual  environment  should  allow  reading  a  newspaper  subheadlines  at 
arms  length.  High  Definition  Television  (HDTV)  would  pass  this  test. 

The  use  of  moving  base  simulators  to  improve  the  validity  of  locomotion  teleoperations  is  dis¬ 
cussed,  and  data  are  shown  to  support  the  following  recommendations.  Linear  motion  bases  are  not 
needed  if  the  real-world  task  entails  less  than  about  l.l  Gy  of  transverse  acceleration.  If  the  rms  Gy 
is  greater,  it  is  more  important  to  maintain  correct  motion-cue  phasing  (<30  degree  disparity)  than 
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amplitude,  which  can  be  cut  to  about  1/5  the  actual  levels.  Rotary  motion  rates  aie  sensed  and  used 
as  “rotary  motion  dampers”  when  the  real-world  task  is  regulation  against  inertial  motion  distur¬ 
bances  (as  in  running),  but  are  ignored  when  the  task  involves  pursuit  of  a  maneuvering  target. 
Details  are  discussed  in  the  presentation. 

Some  rules-of-thumb  for  good  “feel-system”  simulation,  such  as  for  control  manipulanda  are 
given.  Their  vernacular  version  dictates  that  “good”  simulation  of  a: 

-  zero  inertia  and  force-free  control  should  “feel  like  a  stick  of  balsa  wood”  (negligible 
friction,  jerk  or  jitter) 

-  hard  stop  should  “feel  like  a  brick  wall”  (no  creep  or  sponginess) 

-  coulomb  friction  should  “feel  like  sliding  a  refrigerator  magnet”  (remain  in  place;  no  creep, 
bounce  or  jitter) 

-  centering  detent  should  “yield  an  audible  “klunk”  when  traversed”  (no  lag  or  sponginess) 

Try  these  on  your  favorite  manipulandum! 

Finally,  simulation  tests  of  teleoperators  and  virtual  environments  should  include  three  types  of 
measures;  system  performance,  operator  (or  robot)  “behavior”  (e.g.,  describing  functions,  etc.),  and 
mental  workload  evaluations.  The  criteria  for  adequate  mental  workload  measures  are  that  they  be: 

1.  Relevant  -  to  mental  workload,  per 

2.  Sensitive  -  monotonic  correlation  with  workload 

-  insensitive  to  irrelevant  variables 

-  high  test  power  (covariance/residual) 

3.  Concordant  -  ubiquitous  trends  in  target  population 

4.  Reliable  -  proven  test;  retest  repeatability 

-  proven  norms  and  statistics 

5.  Convenient  -  easy  to  administer  and  learn 

-  portable 

-  low  cost  for  adequate  level  of  reliable  result 

Properly  designed  subjective  ratings  usually  fare  better  than  such  measures  as  evoked  response 
potentials!  Examples  of  the  foregoing  approach  are  given,  in  optimizing  motion  washouts. 

If  your  teleoperator  or  virtual  environment  simulation  can  pass  these  rule-of-thumb  tests,  then 
you  can  be  fairly  sure  that  it  will  provide  results  which  can  stand  the  final  test  of  real-world  valida¬ 
tion,  form  a  proper  data  base  for  scientific  modeling  and  engineering  design,  and  be  liked  by  its 
users. 
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CONTROL  BW  AND  H.Q.  DELAYS  ARE  RELATED 
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DOMINANT  CUES 
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IMPUCATIONS  -  VISUAL  DELAYS 

.  optimized  control  tasks  have  BW/t'^  «  1.0 

•  Human  will  adaf}t  Tg^  to  »  1/5  BW,  down  to  t  » .20  sec 

•  For  high  BW  tasks:  BW/t’^  »  1.0  and  >  .16  sec 

.  Control  theory  shows  that  any  simulation  delay  artifacts  must  be  «.20 
sec  to  provide  "valid"  simulation 

Criterion  1 

•  A  "Valid"  simulation  requires  delay  artifacts  be  <c  .20  sec: 

<.02  sec  {60Hz)  =  Excellent;  <.04  sec  =  Good;  >.10  sec  =  Invalid 
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FIELX)  OF  VIEW  VS.  KINETOSIS 

Fixed-Base  Vertigo  vs.  Reid  of  View 


PratobilitY  ot  Vtfino  (SlMrl*T«nii*Awartn«s*l 


Implications:  For  wide  FUV  use  "impoverished  scene" 

Criterion  2:  Keep  FOV  to  cover  less  lateral  view  than  a  curve  of  ±.10  would 
cover  at  real-world  speeds  " 

Other  factors: 

•  More  parafoveal  details  ^  worse  conflict 

•  Proper  framing  of  parafoveal  streamers  is  important 


VISUAL  UPDATE  AND  RESOLUTION  OF  DISPUVYS 


Criterion 

•  For  "inner  loop"  control  of  path  rate  and  curvature,  "streamers"  from 
scene  microtexture  should  not  jerk  or  strobe  backward. 

(«  20-60  frames/sec  update) 

Criterion 

•  For  "outer  loop"  guidance  and  navigation  must  be  able  to  perceive  and 
understand  cues  at «  3-5  sec  ahead  of  vehicle 

Implies  «  2  min-arc  of  resolution  for  signs,  potholes,  targets  (e.g.,  HDTV) 


Criterion 

•  For  Virtual  Environment  should  be  able  to  read  a  newspaper  at  arm’s 
length.  (Subheadlines  =  OK)  (e.g.,  HDTV) 
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VERIFICATION  OF  VALID  SIMULATION 


46 


Cooper-Harpar  RaUnga: 
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MAN-IvIACHINE  INTERFACES  IN  HEALTH  CARE 


Steve  Chiirles,  M.D.  and  Roy  E.  Williams 
Center  for  Engineering  Applications 
Memphis,  Tennessee 


The  surgeon,  like  the  pilot,  is  confronted  with  an  ever  increasing  volume  of  voice,  data  and 
image  input.  Simultaneously,  the  surgeon  must  control  a  rapidly  growing  number  of  devices  to 
deliver  care  to  the  patient.  The  broad  disciplines  of  man-machine  interface  design,  systems  integra¬ 
tion  and  teleoperation  will  play  a  role  in  the  operating  room  of  the  future.  The  purpose  of  thio  com¬ 
munication  is  to  report  the  incorporation  of  these  design  concepts  into  new  surgical  and  laser  deliv¬ 
ery  systems.  A  review  of  each  general  problem  area  and  the  systems  under  development  to  solve  the 
problems  foliows. 

Ophthalmology  is  a  machine-dependent  specialty  embracing  the  patient-machine  interface  and 
the  surgeon-machine  interface.  As  many  as  fifteen  independent  systems  may  be  used  in  the  operating 
room  during  a  surgical  procedure.  Each  system  is  currently  manually  controlled  creating  an  intensive 
operator  interface  environment.  The  surgical  operating  system  known  as  the  Ocular  Connection 
Machine  (OCM)  has  reduced  the  number  of  stopcocks,  foot  pedals,  adapters,  displays,  controls  and 
pow^  cords  by  several  orders  of  magnitude.  Physiological  analog  parameters,  including  intraocular 
temperature  and  pressure,  previously  monitored  with  human  interaction  are  now  controlled  by  servo 
systems,  while  surgical  analog  parameters,  i.e.,  cutting  and  thermal,  are  controlled  by  dual  propor¬ 
tional  positioning  of  the  surgeon’s  foot.  Multiple  displays  have  been  replaced  by  one  thus  reducing 
viewing  time  and  errors.  By  reducing  the  amount  of  man-machine  interactions,  the  system  markedly 
reduces  set-up  time  and  clean-up  time,  as  well  as  improving  reliability,  functionality  and  patient 
safety. 

Laser  treatment  of  tl.e  retina  is  an  intensive  man-machine  task.  The  ophthalmologist  must 
determine  the  area  of  treatment,  aim  the  beam  and  fire  while  the  patient  is  awake.  The  Laser  Imaging 
Work  Station  (LIWS)  currently  under  development  alleviates  much  of  the  ophthalmologist-machine 
interaction  and  reduces  the  discomfort  of  the  patient-machine  interaction.  The  LIWS  is  an  integrated 
system  containing  a  high-speed  digital  correlation  retinal  tracker,  an  automated  laser  delivery  system 
which  controls  laser  power,  wavelength,  spot  size,  duration  and  location,  and  an  expert  system  with 
algorithms  to  avoid  the  macula,  optic  nerve,  vessels  and  previous  photocoagulated  areas.  Surgeon 
interface  occurs  off-line  whereby  a  patient  treatment  template  is  determined  and  down-loaded  to  the 
laser  delivery  system. 

Real-time  diagnostic  imaging  of  a  patient’s  eye  is  currently  resolution  limited,  two-dimensional 
and  user  interaction  intensive.  A  system  is  currently  under  development  which  provides  quasi-real 
time  stereo  (3D)  ultrasound  and  improved  lateral  resolution.  Depth  of  the  viewing  volume  within  the 
eye  (focal  plane)  is  identical  to  current  focus  mechanisms  used  by  ophthalmologists  whereby  a  hand¬ 
held  lens  is  moved  forward  and  backward  yielding  different  viewing  planes  (indirect 
ophthalmoscopy).  This  hand-eye  coordination  and  resulting  spatial  information  will  be  natural  for 
ophthalmologists. 
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Utilization  of  advanced  robotic  systems  in  a  teleoperation  environment  will  enable  microsur¬ 
geons  to  improve  present  operations  and  allow  those  surgeons  not  as  dexterous  to  perform  operations 
not  manageable  with  present  systems.  For  example,  epiretinal  membranes  too  thin  to  remove  without 
risk  of  macular  holes  would  be  a  realizable  target  for  the  ophthalmic  microsurgeon.  By  applying 
6  degrees-of-  freedom  in  a  coarse-fme  strategy  with  translational  resolutions  of  10  to  20  microns, 
these  techniques  and  others  are  made  possible.  The  surgeon  controls  the  bilateral  robotic  system 
through  the  surgeon-machine  interface  (SMI)  which  provides  stiffness  feedback  by  sensing  force  and 
position  at  the  operating  site  and  backdriving  actuators  within  the  SMI.  Further  sensing  of  optical, 
electrical  and  mechanical  impedance  at  the  operation  site  is  fed  back  to  the  surgeon  via  the  foot 
pedal  interface,  offering  another  realm  of  sensory  feedback. 
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TELEROBOTICS  IN  REHABILITATION:  BARRIERS  TO  A  VIRTUAL 

EXISTENCE 


Larry  Leifer,  Machiel  Van  Der  Loos,  and  Stefan  Michalowski 
Stanford  Center  for  Design  Research 
Stanford,  California 


The  Need  for  Telerobotics  in  Rehabilitation 

There  are  over  67,500  quadriplegics  in  the  United  States  today,  with  an  estimated  2,400  to  4,000 
new  injuries  occurring  each  year  (Stover  et  al.,  1986).  These  injuries  occur  most  frequently  among 
young  males  between  the  ages  of  16  and  30.  Due  to  advances  in  medical  treatment  (antibiotics  and 
skin  care),  these  individuals  now  have  a  relatively  normal  life  expectancy.  They  are  alive,  but  dra¬ 
matically  cut  off  from  most  aspects  of  a  normal  existence.  They  live,  virtually,  through  the  actions  of 
others.  Perhaps,  life  would  be  more  fulfilling  if  they  were  also  offered  the  opportunity  to  directly 
control  their  environment  through  telerobotic  tools  for  independent  living.  Virtual  telerobotic  envi¬ 
ronments  for  the  rest  of  us  could  bring  reality  to  these  individuals. 

It  is  estimated  that  caring  for  a  quadriplegic,  including  standard  medical  treatment,  equipment 
maintenance  and  attendant  care,  costs  about  $47,000  (1980  dollars)  per  year.  This  translates  to 
approximately  1.9  million  dollars  over  forty  ye'^rs  (Kalsbeek  et  al.,  1980).  The  net  direct  cost  to  the 
Department  of  Veterans  Affairs  will  be  approximately  $5  billion  for  its  current  population  of  service 
connected  quadriplegics.  On  the  promising  side.  Young  et  al  (1982)  estimated  that  every  dollar  spent 
for  rehabilitation  research  and  development  returns  $1 1  in  cost  benefits  to  society.  In  the  case  of 
quadriplegia,  the  cost  of  attendant  care  (nominally  $25/hr  to  the  insurer  and  increasing)  can  be 
reduced  by  providing  personal  telerobots  (nominally  $5/hr  and  decreasing).  Hammel  et  al.  (1989) 
demonstrated  that  telerobots  can  satisfy  the  vocational  manipulation  needs  of  personal  computer 
users  for  periods  of  over  four  hours  at  a  time  without  attendant  intervention.  Employment  makes  it 
possible  to  recover  some  or  all  of  the  indirect  cost  of  severe  disability. 

Barriers  to  Telerobotics  Tecbnoiogy  in  Rehabilitation  and  Health  Care 

If  the  direct  cost  of  severe  disability  is  so  high,  and  telerobotics  technology  is  available  to  help 
reduce  costs,  then  what  have  been  the  barriers  to  its  widespread  acceptance  and  deployment?  Clini¬ 
cal  experience  with  telerobots  suggests  that  there  are  several  key  barriers: 

Social  Barriers:  As  a  society,  we  place  little  emphasis  on  restoration  of  function  for  persons 
with  disability,  we  prefer  to  “take  care  of  them”.  Because  the  economics  of  cost  and  ben¬ 
efit  are  not  coupled,  we  fail  to  see  the  opportunity.  However,  even  if  we  began  deploy¬ 
ment  today,  our  society  has  educated  too  few  persons  to  support  the  advanced  assistive 
devices  (i.e.,  we  have  enough  researchers  to  create  independent  living  tools,  but  too  few 
development  and  service  persons  to  support  clinical  and  domestic  usage).  These  are  the 
dominant  factors  impeding  wider  adoption  of  advanced  technical  aids  for  persons  with 
disability. 
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Institutional  Barriers:  Government  programs  are  scattered  and  disjoint.  There  is  no  system¬ 
atic  vision  and  considerable  inter-agency  protectionism.  Perhaps  most  significantly,  the 
“conflict  of  interest”  witch  hunt  in  government  makes  it  virtually  impossible  to  transfer 
laboratory  results  to  the  commercial  sector  in  a  timely  and  cost-effective  manner.  For  its 
part,  the  commercial  sector  does  not  take  the  investment  risk  to  develop  these  devices 
because  it  is  not  clear  that  third  party  reimbursement  will  be  forthcoming. 

Technical  Barriers:  Here,  at  least,  the  science  and  engineering  communities  do  have  some 
control  of  the  issues,  these  include; 

The  human-machine-interface  is  the  dominant  technical  barrier  to  widespread  use 
of  telerobots  in  rehabilitation.  Text,  voice,  graphic  and  kinematic  command-con¬ 
trol  interfaces  are  very  cumbersome  for  robot  motion  specification,  planning  and 
supervision.  This  forces  the  user  to  be  overly  dependent  on  pre-programmed 
motions  and  the  technicians  who  create  them.  We  must  v'ork  towards 
“instructable”  telerobots. 

The  machine-environment-interface  is  the  second  most  deficient  aspect  of  teler¬ 
obotics  in  rehabilitation.  The  absence  of  sensor  driven  grasp  and  object  approach- 
avoidance  reflexes  forces  the  user  to  directly  control  end-effector  motion  under 
difficult  observational  circumstances  and  without  “natural  kinesthetic  cues”.  We 
must  develop  robust  sensate  end-effectors  and  the  “reflexes”  to  make  them  useful. 
Force  (impedance)  control  will  be  a  requirement  for  advanced  user  support  tasks. 

Mobility,  or  the  lack  of  it,  defines  the  telerobot’s  work  space  and,  in  part,  its  ultimate 
utility.  One  can  not  reasonably  expect  general  cost-effectiveness  when  people 
must  be  available  to  bring  work  to  the  telerobot.  Raw  mobility  is,  however,  not 
enough.  Remote  presence  makes  much  greater  demands  on  the  user-interface  and 
telerobotic  sensory  capability.  A  more  “intelligent”  robot  may,  in  fact,  be  the 
greatest  challenge  yet  to  the  user-interface  designer. 

Fault-intolerance  is  an  overriding  shortcoming  of  almost  all  robots.  As  program¬ 
mable  electro-mechanical  systems,  they  are  inherently  subject  to  a  very  wide 
variety  of  fault  modes.  Not  even  the  digital  controller  in  current  machines  take 
advantage  of  computer  fault-tolerance  architectures  (which  themselves  make  no 
provision  for  sensor  and  actuator  failure  modes).  Widespread  personal  use  of 
telerobots  will  require  fundamental  progress  in  design  for  fault-tolerance  (we  must 
get  well  beyond  just  being  careful). 
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Telerobotics  in  Rehabilitation:  Once  Around  the  World 


A  small  number  of  underfinanced  telerobotics  teams  around  the  world  are  attempting  to  over¬ 
come  these  barriers.  The  most  recent  compilation  of  papers  are  in  the  Proceedings  of  the  “First  Inter¬ 
national  Workshop  on  Robotic  Applications  in  Medicine  and  Health  Care”  (1988).  The  next  edition 
will  grow  out  of  the  International  Conference  on  Rehabilitation  Robotics  (sponsored  by  the  A.  I. 
duPont  Institute,  June,  1990).  The  following  synopses  highlight  some  of  the  ongoing  R&D  (see 
Table-4  for  a  technical  comparison  of  the  telerobots  used. 

The  Johns  Hopkins  University  Applied  Physics  Laboratory  (JHU/APL)  project  (Schneider  et  al., 
1981)  concentrated  on  the  implementation  of  a  workbench-mounted  robot  intended  to  perform 
activities  of  daily  living  (ADL)  tasks.  The  design  (initially  4  degrees-of-freedom  then  extended  to  5) 
was  derived  from  prosthetic  arm  technology.  Control  is  accomplished  by  a  head-motion  (chin)  con¬ 
trolled  joystick  for  joint  specific  motion.  Pre-programmed  motions  are  invoked  by  menu-selection 
and  input  command  via  a  sip-and-puff  switch. 

The  Tufts-New  England  Medical  Center  robotics  project  (Gilbert  and  Foulds,  1987;  now  at  the 
A  I  duPont  Institute)  concentrated  on  the  design  of  a  universal  robot  programming  language, 
CALVIN,  to  provide  a  common  interface  to  the  many  different  manipulators  available.  Using 
CALVIN,  they  set  up  a  variety  of  small,  low  cost,  robot  work  cells  (typically  4  and  5  degree-of- 
freedom  manipulators)  in  clinical  rehabilitation  settings  for  disabled  children.  The  clinical  objective 
is  to  foster  intellectual  development  of  the  child. 

The  Boeing  Company  developed  a  voice-controlled  workstation  using  the  Universal  Machine 
Intelligence  RTX  (5  degree  of  ft'eedom)  manipulator  (Fu,  1986).  The  distinctive  feature  of  this  pro¬ 
ject  is  that  it’s  user  interface  is  a  voice  query  system  for  large  data  bases  developed  by  Boeing.  Prab 
Command,  Inc.,  began  marketing  the  system  in  1988  for  vocational  applications.  There  is  very  little 
telerobotic  function  beyond  pre-programmed  diskette  loading.  The  internal  project  at  Boeing  has 
ended  and  Prab  sold  the  technology  to  the  Zenith-Heath. 

From  1980  through  1988,  the  Veterans  Administration  sponsored  a  collaborative  effort  with 
Stanford  University  (Leifer,  1982;  Hammel  et  al.,  1989)  to  test  the  hypothesis  that  industrial  manipu¬ 
lators  could  cost  effectively  serve  the  needs  of  severely  impaired  spinal  cord  injury  patients.  The 
project  demonstrated  that,  through  voiced  commands,  the  utility  and  reliability  of  a  high  perfor¬ 
mance  manipulator  (PUMA-260)  and  control  language  (VAL-2)  can  yield  attractive  cost-to-benefit 
performance  ratios  when  the  telerobot  is  able  to  operate  for  four  hours  without  attendant  interven¬ 
tion.  Sensate  end-effector,  mobility,  3D  point  designation  and  natural  language  studies  laid  the  foun¬ 
dation  for  further  R&D.  A  desktop  vocational  assistant  robot  (DeVAR)  version  of  the  system  is 
available  commercially. 

Outside  the  United  States,  the  Canadian  Neil  Squire  Foundation  has  developed  a  low-cost 
manipulator  designed  for  desktop  applications  in  rehabilitation  (Cameron,  1986).  The  system  is 
being  sold  by  the  foundation.  At  the  Institute  for  Rehabilitation  Research  in  the  Netherlands,  Hok 
Kwee  (previously  with  the  French  Spartacus  project  (Kwee,  1983)  and  his  colleagues  have  devel¬ 
oped  a  wheelchair  mounted,  joystick-controlled,  manipulator  (MANUS,  Kwee,  1987).  They  are 
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particularly  interested  in  collocation  of  the  robot  and  user  at  all  times.  The  manipulator  is  expected  to 
function  as  an  arm,  not  really  a  telerobot. 

While  human  attendant  care  is  the  norm  today,  and  telerobotics  technology  promises  to  release 
some  attendant  time  cost  effectively,  there  is  a  third  alternative.  Capuchin  monkeys  have  been 
trained  to  provide  attendant  services  from  feeding  to  appliance  operation  (internal  VA  program 
review,  1988).  Monkey  training  is  expensive,  labile,  and  done  by  methods  objectionable  to  the  ani¬ 
mal  rights  community.  Program  retention  (to  a  maximum  of  12  short  tasks)  and  willfulness  remain 
significant  obstacles  to  the  monkey  assistant  concept.  The  approach  is  particularly  limited  by  the  fact 
that  monkeys  can  only  work  one-on-one  with  their  user.  The  distractions  of  institutional  and  voca¬ 
tional  settings  render  the  monkey  ineffective. 

Table  1  presents  an  extended  comparison  of  the  strengths,  capabilities  and  limitations  of  five 
approaches  to  augmenting  the  independence  of  severely  impaired  persons.  Supported  by  a  growing 
body  of  experimental  evidence,  this  comparison  strongly  suggests  that  telerobots  can  become  cost- 
effective  personal  and  vocational  assistants.  Table  2  lists  many  of  the  feeding,  personal  hygiene, 
vocational,  and  recreational  tasks  that  have  been  demonstrated  over  the  past  8  years  f'y  four  genera¬ 
tions  of  the  Stanford-VA  Rehabilitation  R&D  Center’s  Telerobotic  systems.  Table  3  ’dentifies  the 
technical  functions  required  to  perform  these  tasks. 

A  Partial  View  of  the  Future 

The  operator  interface  will  remain  the  dominant  problem.  It  is  so  difficult  that  most  systems 
designers  prefer  to  ignore  it  and  focus  on  “tangible”  technical  specifications.  It  is  likely  to  use  mul¬ 
tiple-channels  in  the  sense  of  incorporating  a  “natural  robot  instructional  language”  and  bi-direc¬ 
tional  pictographic  dialog  between  the  operator  and  the  system.  Increasingly  “autonomous”  teler¬ 
obots  will  actually  increase  the  burden  on  this  interface. 

Sensor  driven  motion  planning  and  autonomous  grai^  avoidance  reflexes  will  become  common¬ 
place.  The  rate  of  introduction  of  such  features  will,  however,  be  much  slower  than  expected.  In  part, 
this  is  due  to  the  fact  that  system  architectures  will  continue  to  be  “fault-intolerant”  such  that  the 
introduction  of  both  sensors  and  programmed  reflexes  will  bring  new  reliability  problems  with  them. 

Force  (impedance)  control  of  telerobots  will  continue  to  evolve  slowly  even  thought  this  capa¬ 
bility  is  a  fundamental  requirement  for  any  robot  that  must  work  intimately  with  humans.  Physical 
therapy  by  robots  is  both  needed  and  impossible  without  force  control.  These  lines  of  technical  evo¬ 
lution  will  themselves  depend  on  getting  more  applications  feedback  from  telerobots  in  the  clinic, 
home  and  office.  In  combination,  we  see  a  rather  daunting  challenge. 
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Table  la  In-depth  Review  of  the  Capabilities,  Strengths  and  Weaknesses  of  Five  Basic  Approaches 
to  the  Restoration  of  Manipulative  Function  for  People  with  Severe  Physical  Limitations  (none  = 
limited  =  *,  usable  =  **,  good  =  ***,  excellent  =  ****). 


Environment  Controllers 

I^sthetics  Workstations  -  Seamone 
Monkeys  -  Willard 


Robots  - 

Leifer 

People 

Human  Input/Output  Factors 

COMMAND 

* 

* 

**** 

Substitution  errors 

5% 

5% 

10% 

10% 

1% 

Detection  errors 

10% 

10% 

25% 

15% 

2% 

Number  of  commands 

10 

20 

12 

100 

2000 

Syntax  options 

- 

menu 

menu 

program 

English 

User  programming 

- 

- 

- 

yes 

yes 

CONTROL 

- 

- 

- 

- 

Real-time 

- 

- 

- 

yes 

- 

Degrees  of  freedom 

- 

- 

- 

7 

- 

Rexible 

- 

- 

- 

yes 

- 

User  programmable 

- 

- 

- 

yes 

- 

DIALOG 

» 

* 

*** 

**** 

Feedback  modes 

lights 

lights 

noises 

voice 

voice 

Explanations 

- 

- 

- 

status 

unlimited 

Inference 

- 

- 

limited 

unlimited 

Rule  based 

- 

- 

- 

limited 

general 

User  adaptive 

- 

- 

7 

limited 

unlimited 

Model  based 

- 

- 

? 

sensory 

general 

Machine  Input/Output 

AUTONOMOUS  PLANNING 

• 

- 

- 

** 

**** 

Path 

- 

- 

- 

limited 

general 

Strategy 

- 

- 

- 

limited 

general 

Data  driven 

- 

- 

- 

limited 

general 

PROGRAMMABLE  REFLEXES 

- 

- 

* 

** 

**** 

Force  compliance 

- 

- 

** 

** 

Contour  following 

- 

- 

- 

«* 

Proximity  sensing 

- 

- 

- 

*** 

- 

Collision  avoidance 

- 

- 

* 

**** 

POSmON/ORIENTATION 

- 

** 

*** 

**** 

Degrees  of  freedom 

- 

4 

9 

7 

9 

Radius  of  working  volume 

- 

40cm 

30cm 

40cm 

55cm 

Precision 

- 

low 

low 

high 

flexible 

Repeatability 

- 

2mm 

3mm 

0.2mm 

3mm 

Strength 

- 

low 

low 

low 

flexible 

Speed 

- 

low 

moderate 

flexible 

flexible 

MOBILITY 

- 

- 

** 

**** 

Degrees  of  freedom 

- 

- 

6 

4 

6 

Range 

rooms 

desk 

room 

rooms 

unlimited 

Remote  control 

IR  link 

- 

voice 

voice 

voice 

Autonomous 

- 

- 

7 

limited 

unlimited 

GRASP 

- 

♦ 

*« 

**** 

Degrees  of  freedom 

- 

1 

6 

1 

6 

Grip  force 

- 

1kg 

i.5kg 

3kg 

25kg 

Dexterity 

- 

minimal 

good 

minimal 

excellent 

SENSATION 

- 

- 

** 

**** 

Tactile 

- 

- 

**** 

Force 

- 

- 

**** 

Proximity 

- 

- 

- 

♦  * 

- 

Vision 

- 

- 

- 

Audition 

- 

- 

**** 

- 
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Table  lb 


Assessment 

PERFORMANCE 
Task  time 
Training  time 
Number  of  tasks 
Commands  per  task 
Precision 
Repeatability 
Reliability 
Accessibility 
SAFETY 

Intention  errors 
Intrusion  errors 
Contact  errors 
COST 

Hardware 
Software 
Training 
Maintenance 
Cost/hour  of  use 


Environment  Controllers 

Prosthetics  Workstations  -  Seamone 
Monkeys  -  Willard 


Robots  - 

Leifer 

People 

** 

** 

*** 

*** 

seconds 

minutes 

minutes 

minutes 

seconds 

2hrs 

30hrs 

20hrs 

40hrs 

4hrs 

10-20 

15-30 

10-20 

20-200 

40-400 

1 

1-4 

1-4 

1-20 

1-2 

- 

5mm 

2mm 

0.2mm 

2mm 

- 

1mm 

3mm 

0.2mm 

3rmn 

high 

good 

fair 

good 

fair 

24hrs 

24hrs 

12hrs 

24hrs 

8hrs 

**** 

*** 

** 

** 

***♦ 

5%  ■ 

10% 

5% 

10% 

4% 

0% 

4% 

5% 

5% 

2% 

0% 

2% 

4% 

2% 

1% 

**** 

** 

** 

** 

♦ 

low 

medium 

medium 

high 

low 

low 

low 

high 

high 

low 

low 

medium  high 

medium 

low 

low 

medium 

medium 

low 

high 

$l/hr 

$4/hr 

$  10/hr 

$5/hr 

$25/hr 
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Table  2  Meal  preparation  and  service,  vocational  material  handling,  personal  hygiene,  and  recre¬ 
ational  tasks  that  have  been  performed  for  and  with  disabled  individuals  across  four  generations  of 
desktop  robotic  assistants  (DeVARs  I,  II,  III,  IV) 


Meal  preparation  and  service 

Vocational  material  handling 

Personal  hygiene 

arrange  table  setting 

write  with  pen  and/or  pencil 

wash  and  dry  face 

open  and  close  microwave  door 

retrieve  books  and  manuals 

brush  teeth 

open  and  close  refrigerator  door 

set  up  books  for  reading 

dispense  toothpaste 

manipulate  containers 

turn  book  and  report  pages 

use  electric  shaver 

set  appliance  timer 

turn  on-off  computer  equipment 

retrieve  mouthstick 

pour  liquids  and  solids 

type  on  keyboard 

comb  and  brush  hair 

beat  eggs 

adjust  keyboard  position 

blow  nose 

toss  salad 

operate  private  and  speaker  phones 

apply  makeup 

soup  preparation  and  service 

insert  and  retrieve  diskettes 

heat  and  serve  casseroles 

insert  and  retrieve  audio  tapes 

serve  pudding 

operate  dictation  equipment 

serve  fruit 

manipulate  printout 

prepare  and  serve  spaghetti 

voiced  control  of  generic  software 

prepare  and  bake  a  cake 

load  and  operate  printer 

use  knives,  forks  and  spoons 

retrieve  and  serve  medication 

retrieve  drinks 

circuit  boards  for  inspection 

mix  drinks 

operate  electronic  control  units; 

lock  and  unlock  cabinet  doors 

door  operation 

light  and  extinguish  candles 

security  system 

light  and  extinguish  cigarette 
open  and  close  storage  drawers 

stereo  equipment 

room  lights 

Recreation 

window  open  and  close 

paint  and  sketch 

arrange  flowers 

hand  out  flowers 

hand  out  candy  and  souvenirs 

perform  one  armed  ballet 

checkers 

monopoly 

television 

operate  video  games 

pac-man 

invaders 

play  board  games; 
chess 
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Table  3.  Several  functional  capabilities  are  needed  in  a  Telerobotic  Assistant  to  achieve  utility  in  the 
performance  of  independent  living  tasks,  not  needed  ( ),  should  have  (□),  must  have  (■) 


MAINTENANCE  Tasks 

Food  Preparation 
Food  Service 
Personal  Hygiene 
Personal  Grooming 
Clothing  Management 
Appliance  Usage 
VOCATION  Tasks 
Storage  &  Retrieval 
Equipment  Operation 
Assembly 
Word  Processing 
Computing 
Materials  Processing 
RECREATION  Tasks 
Reading 
Film  &  Video 
Performing  Arts 
Graphic  Arts 
Sports 
Social  Interaction 
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VIRTUAL  ENVIRONMENTS 


CREATING  BETTER  VIRTUAL  WORLDS 


Thomas  A.  Furness,  III 
University  of  Washington 
Seattle,  Washington 


INTRODUCTION 


Computing  machines  have  brought  new  opportunities  as  well  as  challenges  to  our  information 
age.  While  computer  capacities  and  speeds  have  increased  remarkably,  our  ability  to  communicate 
with  these  information  engines  is  still  llrrtited  by  poor  interfaces  between  the  human  and  the  com¬ 
puter.  To  harness  and  exploit  the  power  of  the  computer  as  a  tool  and  extender  of  our  intellect,  it  is 
essential  that  we  build  new  interfaces  which  are  intuitive,  transparent  and  take  advantage  of  our 
innate  spatial  abilities. 

The  new  technology  of  virtual  interfaces  provides  a  revolutionary  way  to  improve  the  coupling 
of  the  human  to  the  computing  machine.  Virtual  controls  and  displays  allow  panoramic  presentations 
in  three  dimensions  to  be  made  to  the  eyes,  ears  and  hands  of  the  user  by  projecting  “virtual”  images 
into  the  senses.  To  gain  this  global  experience  “the  human  literally  wears  the  computer.”  Virtual 
interfaces  transfer  stimuli  to  the  eyes,  ears  and  hands,  and  use  head,  eye,  hand  movement  and  speech 
to  control  the  machine.  In  this  way  the  operator  can  be  surrounded  by  a  “cir:  cm  ambience”  of  com¬ 
puter  synthesized  information  which  becomes  a  spatial  world  through  which  ne  and  machine  can 
effectively  communicate.  The  operator,  in  turn,  interacts  with  his  global  medium  by  looking  at 
objects,  pointing  his  hands  and  giving  verbal  commands.  The  medium  also  allows  virtual  objects, 
which  appear  to  be  real  but  which  are  virtual  projections,  to  be  touched  and  manipulated  by  the 
hands.  In  order  to  create  the  virtual  space  representation  of  information,  the  virtual  display  hardware 
components  are  programmed  with  “mindware.”  Mindware  is  a  special  class  of  software  that  takes 
into  account  the  perceptual  organization  <^f  the  human  and  dynamically  creates  three-dimensional 
sound,  video  and  tactile  images  which  appear  as  an  artificial  reality  surrounding  the  user. 

The  virtual  interfaces  described  above  not  only  solve  many  existing  interface  problems  but 
empower  new  and  novel  interfaces  for  operators  in  aerospace  vehicles  and  other  applications  such  as 
teleoperation,  computer-aided  design,  medical  imaging,  prostheses  for  the  handicapped  and  enter¬ 
tainment.  Even  though  some  aspects  of  virtual  interface  technology  have  been  under  development 
for  two  decades,  there  are  still  several  key  problems  which  must  be  addressed  and  resolved  to  make 
these  concepts  practical  for  research,  industrial  and  consumer  applications.  Tliese  problems  are 
summarized  below: 
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THE  PROBLEMS 


1.  We  don’t  understand  the  human  factors  of  virtual  space. 

2.  We  don’t  know  how  to  measure  the  “goodness”  of  virtual  worlds. 

3.  Except  for  the  more  expensive  military  versions,  virtual  displays  lack  sufficient  resolution  for 
wide  field-of-view  presentations. 

4.  Position  sensing  technology  (for  control  and  image  stabilization)  needs  improvement  to 
increase  precision  and  update  rate  while  reducing  noise  and  sensitivity  to  conductive  materials. 

5.  Current  graphics  engines  have  limited  update  rate  and  polygon  count. 

6.  The  headgear  is  too  heavy. 

7.  There  are  too  many  /ires. 

8.  There  is  a  need  for  a  universal  software  architecture  and  tool  kit  for  supporting  future  devel¬ 
opment  and  application  of  virtual  worlds. 

9.  Virtual  interfaces  are  too  expensive. 

10.  There  is  no  forum  currently  for  discussing  the  development  and  applications  of  virtual  worlds. 

POSSIBLE  SOLUTIONS 


As  research  and  development  laboratories  in  universities  and  industry  respond  to  the  growing 
excitement  about  virtual  worlds,  we  would  propose  that  a  research  agenda  be  established  to  solve  the 
problems  above.  Here  are  recommended  approaches: 

1 .  Establish  a  national  knowledge  base  regarding  the  ergonomics,  technology  and  application 
of  virtual  interfaces.  Build  a  centralized  literature  collection,  and  interface  with  the  Crew  System 
Ergonomics  Infonnatiou  Center  (CSERIAC)  at  Wright-Patterson  AFB,  OH.  Add  to  the  Handbook  of 
Perception  and  Human  Performance  and  the  Engineering  Data  Compendium  published  by  the  Air 
Foice.  Begin  a  scholarly  journal  to  address  issues  of  virtual  interfaces  including  the  psychophysics, 
technologies  and  applicauons  of  virtual  worlds. 

2.  Explore  new  psychophysical  and  neurophysiological  measurement  techniques  which  operate 
within  virtual  worlds. 

3.  Develop  high  resolution  parallel  image  projection  approaches  which  reduce  bandwidth  and 
don’t  require  the  production  of  an  aerial  image. 
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4.  Develop  position  measurement  approaches  which  are  passive. 

5.  Improve  image  stabilization  (close  loop  outside  graphics  processor).  Develop  graphics  pro¬ 
cessor  to  support  parallel  information  input. 

6.  Eliminate  wires  with  passive  sensing  and  data  links. 

7  Create  microscanner  for  Maxwellian  view  image  projection. 

8.  Create  mindware  to  take  into  account  the  perceptual  organization  of  the  human.  Standardize 
on  network  protocols  for  televirtuality  and  transportability  of  software  and  applications. 

9.  Establish  a  special  interest  group  on  virtual  interface  technologies  under  the  aegis  of  the 
ACM,  IEEE  or  HFS. 


CONCLUSIONS 


Virtual  interfaces  provide  a  bold  new  opportunity  for  solving  many  of  the  perplexing  problems 
of  interfacing  human  and  machine  intelligences.  As  pioneers,  we  are  obligated  to  pursue  the  devel¬ 
opment  of  virtual  interface  technologies  in  a  systematic  way  and  to  leave  a  technology  base  and  tools 
as  our  legacy  for  others  to  build  upon.  If  developed  systematically,  virtual  interfaces  can  be  one  of 
the  greatest  advances  of  our  age. 
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INTERACTING  WITH  VIRTUAL  WORLDS 


David  Zeltzer 

Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts 


What  tools  should  we  provide  the  user  for  defining  and  interacting  with  objects  and  agents  in 
virtual  environments  at  varying  levels  of  complexity?  Understanding  the  appropriate  simplifications 
to  make  is  critical  for  modeling  non-trivial  environments  and  agents  with  varying  levels  of  auton¬ 
omy.  I  describe  a  set  of  abstraction  mechanisms  appropriate  for  constructing  and  interacting  with  vir¬ 
tual  worlds,  and  I  discuss  how  programming,  and  direct  manipulation  or  guiding  techniques,  can  be 
used  to  afford  users  interactive  access  to  graphical  simulations  at  appropriate  levels  of  abstraction. 

Structural  abstractions  provide  a  mechanism  for  describing  the  kinematic  structure  of  objects, 
such  as  transformation  hierarchy  for  a  jointed  figure,  and  for  defining  physical  attributes  of  objects, 
e.g.,  mass,  stiffness,  etc. 

Procedural  abstractions  are  a  mechanism  for  defining  processes  that  control  rigid  and  non-rigid 
object  motion,  such  as  elastic  deformation,  collision  detection,  forward  dynamics  or  inverse  kinemat¬ 
ics,  independent  of  the  structure  of  the  agen  or  object. 

Functional  abstractions  are  the  means  with  which  we  associate  procedures  with  objects  and 
object  sub-assemblies,  for  the  purpose  of  defining  meaningful  behaviors.  This  allows  complex, 
largely  unmanageable  systems  with  many  DOFs  -  like  the  human  figure,  with  over  200  DOFs — lO  be 
decomposed  into  a  set  of  small,  manageable  subsystems,  each  with  few  DOFs. 

An  agent  consists  of  a  structural  definition,  a  set  of  functional  units  which  define  the  behavior 
repertoire  of  that  entity,  and  some  means  for  selecting  and  sequencing  (possibly  concurrent)  behav¬ 
iors.  The  mechanism  for  selecting  and  sequencing  these  motor  skills  must  link  action  with  percep¬ 
tion. 

The  lowest  layer  of  abstraction  is  represented  by  objects  and  structures  with  no  procedural 
attachment,  and  is  called  the  machine  level.  At  the  other  extreme,  the  task  level,  guiding  and  pro¬ 
gramming  tools  interact  with  agents  through  their  repertoire  of  behaviors.  Task  level  interaction 
emerges  when  guiding  and  programming  techniques  are  applied  at  higher  layers  of  abstraction. 

Guiding  and  programming  tools  have  a  dual  nature:  they  can  be  used  to  construct  and  modify 
instances  of  any  of  the  abstraction  types;  or  they  can  be  used  at  runtime  as  control  instruments.  I 
describe  how  programming  and  guiding  can  be  used  with  each  of  the  four  classes  of  abstraction. 
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VIRTUAL  REALITY  AT  WORK 


Frederick  P.  Brooks,  Jr. 
University  of  North  Carolina 
Chapel  Hill,  North  Carolina 


1.  INTRODUCTION 

The  utility  of  virtual  reality  computer  graphics  in  telepresence  applications  is  not  hard  to  grasp  and 
promises  to  be  great.  When  the  virtual  world  is  entirely  synthetic,  as  opposed  to  real  but  remote,  the 
utlility  is  harder  to  establish.  Vehicle  simulators  for  aircraft,  vessels,  and  motor  vehicles  are  proving 
their  worth  every  day.  Entertainment  applications  such  as  Disneyworld’s  StarTours  are  technologi¬ 
cally  elegant,  good  fun,  and  economically  viable.  Nevertheless,  some  of  us  have  no  real  desire  to 
spend  our  lifework  serving  the  entertainment  craze  of  our  sick  culture;  we  want  to  see  this  exciting 
technology  put  to  work  in  medicine  and  science. 

2.  TESTING  A  FORCE  DISPLAY  FOR  SCIENTIFIC  VISUALIZATION:  MOLECULAR 
DOCKING  (Ming  Ouh- Young) 

2.1  The  technology.  This  test  uses  a  Model  E-3  Argonne  Remote  Manipulator  (the  ARM) 
that  offers  a  full  6-D  input  position  sensing  and  full  6-D  force  and  torque  output.  A  Sun  4  provides 
computational  power  and  ARM  control;  and  E.&S.  PS-3(X)  provides  the  visual  output.  A  Tektronix 
stereo  plate  is  used  with  passive  circular  polarizing  glasses. 

2.2  The  molecular  docking  application.  The  enzyme  dihydrofolate  reductase  binds  to  a 
variety  of  inhibitors,  natural  and  synthetic.  Of  twelve  inhibitors  studied  by  our  collaborators  at 
Burroughs-Wellcome,  the  precise  structure  of  the  enzyme-inhibitor  complex  is  known  for  six. 

Binding  energy  is  a  complex  function  of  atomic  positions,  involving  Coulomb  forces,  van  der  Waals 
forces,  nuclear  forces,  hydrogen-bonding  forces,  and  thermodynamic  forces.  With  6  degrees  of  free¬ 
dom  in  the  drug  docking,  and  up  to  12  twistable  bonds  in  the  drugs,  the  docker  is  attempting  to  find  a 
global  minimum  of  the  energy  in  a  space  of  up  to  18  dimensions,  a  space  pock-marked  with  local 
minima.  Brute-force  algorithmic  methods  would  require  years  of  computer  time  to  explore  the  space. 

It  is  hard  to  imagine  a  visual  display  that  would  effectively  produce  all  the  needed  insight  to  enable 
a  user  to  manually  dock  the  drug  so  as  to  minimize  energy.  We  postulated  that  feeling  the  forces, 
particularly  the  ( l/r)**7  and  ( l/r)**  1 3  forces  that  define  the  receptor  pocket,  would  substantially 
help  chemi.sts  in  manual  docking. 

2.3  The  expe>‘imental  results.  Chemists  can  dock  an  inhibitor  in  about  25  minutes  using 
our  force  display  system.  Ming’s  controlled  experiments  indicate  an  upper  bound  of  about  a  factor  of 
two  improvement  in  docking  performance  when  the  visual  display  is  aided  with  our  force  display. 
Our  force  display  has  many  primitive  characteristics,  so  there  is  hope  for  some  improvement  in  this 
number,  but  an  order-of-magnitude  improvement  doesn’t  seem  to  be  in  the  cards. 


67 


2.4  Whither  now?  We  plan  to  get  some  actual  drug-design  experience  with  real  drug 
designers.  We  also  want  to  build  or  buy  a  finger-scale,  versus  arm-scale,  6-D  force  output  device. 

The  literature  suggests  that  resolution  should  be  at  least  as  good  with  the  scaled-down  device,  and 
stability  and  ease  of  use  much  better. 

3.  TESTING  A  HEAD-MOUNTED  DISPLAY  FOR  SCIENTIFIC  AND  MEDICAL 
VISUALIZATION  (James  Chung) 

3.1  The  technology.  We  use  the  standard  available  technology:  tiny  TVs  and  Polhemus 
trackers.  Our  PixelPlanes  graphics  engine  is  faster  than  almost  all  of  today’s  delivered  engines.  The 
Polhemus  has  respectable  frame-rate  but  unacceptable  lag  for  head-mounted  displays.  We  measure 
250-400  msec.,  and  lag  seems  to  depend  upon  source-sensor  distance.  We  are  pursuing  several  new 
tracker  technologies. 

3.2  The  only  advantage  of  head-mounted  displays  -  intuitive  navigation.  Considered 
cold-bloodedly,  today’s  head-mounted  displays  have  many  disadvantages:  resolution,  sharpness, 
field-of-view.  Their  inherent  comparative  advantage  is  intuitive  navigation  of  the  virtual  space.  It 
should  be  much  easier  for  the  user  to  know  where  he  is  and  what  he  is  looking  at  than  in  through-the- 
window  graphics.  How  much  is  this  worth?  How  much  does  it  improve  performance  in  a  real 
application? 

3.3  The  radiation-treatment  planning  application.  Radiation  oncologists  have  to  plan  the 
shape,  intensity,  and  direction  of  multiple  beams  of  radiation  so  as  to  bum  the  tumor  without  burning 
other  sensitive  or  vital  organs.  This  task  requires  imaginative  use  of  3-space,  a  good  image  of  the 
anatomy  of  the  patient,  and,  ideally,  real-time  calculation  of  the  dose  to  the  tumor  and  to  unwanted 
organs. 


3.4  The  planned  tests.  We  have  volume-visualized  MRI  anatomical  data  on  tumor  patients. 
Our  collaborating  radiologists,  in  a  controlled  experiment,  will  devise  treatment  plans  using  the 
head-mounted  display  versus  through-the-window  graphics.  We  will  measure  task  time  and 
goodness  of  plan  as  experimental  variables. 
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USING  VIRTUAL  REALITY  AS  A  TOOL  TO  STUDY  THE  RELATIONSHIP 
BETWEEN  HUMAN  BEINGS  AND  PHYSICAL  REALITY 


J.  Lanier 

VPL  Research,  Inc. 
Redwood  City,  California 


This  presentation  focused  on  the  following  areas; 

1 .  VPL’s  Virtual  Reality  system  (the  RB2);  What  it  is  and  what  has  been  learned  from  it. 

2.  Unsolved  technical  problems  in  Virtual  Reality. 

3.  Cultural,  philosophical,  and  political  inquiries  demanded  by  Virtual  Reality. 

4.  The  upcoming  Reality  Net  project. 
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THE  AMES  VIRTUAL  ENVIRONMENT  WORKSTATION: 
IMPLEMENTATION  ISSUES  AND  REQUIREMENTS 


S.  Fisher,  R.  Jacoby,  S.  Bryson,  P.  Stone,  L  McDowall,  M.  Bolas,  D.  Dasaro,  E.  Wenzel, 

C.  Coler,  and  D.  Kerr 
NASA  Ames  Research  Center 
Moffett  Field,  California 


This  presentation  describes  recent  developments  in  the  implementation  of  a  Virtual  Environment 
Workstation  in  the  Aerospace  Human  Factors  Research  Division  of  NASA’s  Ames  Research  Center. 
Introductory  discussions  are  presented  on  the  primary  research  objectives  and  applications  of  the 
system  and  on  the  system’s  current  hardware  and  software  configuration.  Principal  attention  is  then 
focused  on  unique  issues  and  problems  encountered  in  the  workstation’s  development  with  emphasis 
on  its  ability  to  meet  original  design  specifications  for  computational  and  graphics  performance  and 
for  associated  human  factors  requirements  necessary  to  provide  compelling  sense  of  presence  and 
efficient  interaction  in  the  virtual  environment. 

1.  Research  Objectives  and  Applications 

In  the  Aerospace  Human  Factors  Research  Division  of  NASA’s  Ames  Research  Center,  an  inter¬ 
active  Virtual  Environment  Workstation  (VIEW)  has  been  developed  to  aid  in  design,  simulation  and 
evaluation  of  advanced  data  display  and  management  concepts  for  operator  interface  design.  The 
VIEW  system  provides  a  virtual  auditory  and  stereoscopic  image  surround  that  is  responsive  to 
inputs  from  the  operator’s  position,  voice  and  gestures.  As  a  low-cost,  multipurpose  personal  simu¬ 
lation  device,  this  variable  interface  configuration  allows  an  operator  to  virtually  explore  a 
360-degree  synthesized  or  remotely  sensed  environment  and  viscerally  interact  with  its  components. 

At  present,  the  major  thrust  of  this  research  effort  centers  on  the  development  of  guidelines  and 
requirements  that  are  necessary  for  implementation  of  virtual  environments  that  are  matched  as 
closely  as  possible  to  human  cognitive  and  sensory  capabilities  and  that  feature  transparent,  natural 
system  interaction.  It  is  projected  that  design  specifications  derived  from  this  prototype  system  can 
help  define  the  range  of  necessary  and  sufficient  hardware  and  software  architectures  for  future  vir¬ 
tual  environment  systems. 

Application  areas  of  the  virtual  environment  research  are  focused  in: 

TELEPRESENCE:  Development  of  workstations  for  complex  operational  tasks  such  as  teler- 
obotic  and  telepresence  control  of  remotely  operated  robotic  devices  and  vehicles  that  require  a  suf¬ 
ficient  quantity  and  quality  of  sensory  feedback  to  approximate  actural  presence  at  the  task  site. 

DATASPACE:  Development  of  concepts  and  guidelines  for  “portable”  multi-modal  information 
management  systems  such  as  EVA  Spacesuit  visor  display,  with  subsequent  development  of  work¬ 
stations  for  supervision  and  management  of  large-scale  integrated  information  systems  in  which  data 
manipulation,  storage  and  retrieval,  and  system  monitoring  tasks  can  be  spatially  organized. 
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VISUALIZATION:  Design  and  evaluation  of  virtual  environments  to  facilitate  visualization  of 
complex,  three-dimensional  data  structures  in  areas  such  as  architecture,  engineering  and  computa¬ 
tional  fluid  dynamics,  and  for  interactive  training  or  planning  environments  such  as  surgical 
simulation. 

2.  VIEW  System  Configuration:  Performance  requirements  and  implementation 

The  current  Virtual  Interface  Environment  Workstation  consists  of:  a  wide-angle  stereoscopic 
display  unit,  glove-like  devices  for  multiple  degree-of-freedom  tactile  input,  connected  speech 
recognition  technology,  gesture  tracking  devices,  3D  auditory  display  and  speech-synthesis  technol¬ 
ogy,  and  real-time  computer  graphic  and  video  image  generation  equipment.  This  hardware  is  inte¬ 
grated  with  a  realtime  Unix  workstation  that  supports  the  computations  required  to  drive  an  external 
high-performance  realtime  3D  graphics  system,  processes  input  from  up  to  12  realtime  input  periph¬ 
erals  (e.g.,  the  trackers  and  gloves),  and  provides  other  realtime  task  processing.  A  collection  of 
software  called  the  “simulation  framework”  has  also  been  developed  that  consists  of  a  well-docu¬ 
mented  library  of  functions  to  provide  access  to  all  of  the  system  peripherals  and  software  services, 
and  of  a  collection  of  source  files  and  simulation  software  that  demonstrates  the  use  of  the  major 
hardware  and  software  components  that  make  up  the  VIEW  system  in  order  to  facilitate  system 
reconfiguration  for  changing  research  requirements. 

Before  the  beginning  of  the  implementation  phase  of  the  Virtual  Environment  Workstation,  a 
number  of  performance  requirements  and  specifications  for  the  major  VIEW  systems  components 
were  determined.  These  are  discussed  with  respect  to  attaining  an  overall  performance  objective  of 
providing  640x480  pixel  resolution  imagery  at  30  frames  per  second  over  dual,  independent,  syn¬ 
chronized  display  channels  with  image  viewpoints  updated  in  coordination  with  head  and  limb 
motion. 

3.  VIEW  Implementation:  Unique  Issues  and  Problems 

The  VIEW  project  attempts  to  match  media  and  computational  technology  as  closely  as  possible 
to  the  perceptual  and  cognitive  capabilities  of  the  human  operator  in  order  to  achieve  a  state  of 
Telepresence  in  which  a  sufficient  quantity  and  quality  of  sensory  feedback  is  presented  to  the  opera¬ 
tor  to  approximate  presence  at  a  remote  task  site  or  in  a  synthetic  environment.  The  factors  that 
directly  influence  and  effect  the  achievement  of  this  display  conf.guration  are  divided  into  three 
main  areas  and  are  discussed  in  detail  as  examples  of  problems  and  challenges  for  the  development 
of  virtual  environ,  nent  systems. 
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3.1  Computational  issues. 

Problems  in  achieving  VIEW  objective  of  30Hz  frame  rate: 

Overall  cumulative  system  transport  delay 

-  “Realtime”  UNIX 

-  Communication  timing  problems  and  scheduling  artifacts 

-  Problems  with  original  performance  benchmarking 

-  I/O  problems  and  requirements: 

-  RS232  requirements  and  host  modifications 

-  Parallel  communication  between  host  and  graphics  system 

-  Floating  point  accuracy  requirements  in  database  representation. 

-  System  calibration  requirements  for  research  applications  (tracker,  etc.) 

-  The  need  for  evolution  to  object-oriented  software  structure  for  VIEW. 

3.2  Graphics  implementation  issues: 

-  Requirements  and  problems  in  synching  dual  channel  graphics  systems  for  proper 
stereocopic  image  representation: 

-  SGI  IRIS  graphics 

-  HP  SRX/TurboSRX  graphics 

-  ISG  graphics 

Differences  between  human  eye  and  virtual  camera  image  geometry: 

-  Viewport  specification 

-  Perspective  transformations  (world  rotation) 

-  Field  of  View  specification 

-  Display  calibration  requirements 

Matrix/Euler/Quatemian  representation;  difficulties  and  advantages. 

-  Graphics  language  implementations  and  extensions  (GOSAMR) 

-  Object  editor 

-  Problems  with  animating  textures 

-  Needed  extensions 

-  Idiosyncrasies  of  lighting  model  in  VIEW  environment. 

Software  structure  for  3D  information  windows  in  VIEW  environment:  differences 
between  screen  space  and  world  space. 

3.3  Human  Factors  issues  (Telepresence  requirements): 

Guidelines  and  requirements  necessary  to  provide  stereoscopic  images  that  are 
geometrically  correct  representations  of  the  depth  relationships  and  perspective  of  a 
rendered  or  captured  scene  are  discussed  (orthostereoscopy).  For  example,  in  an 
orthostereoscopic  scene,  an  object  in  the  Vinual  Environment  would  subtend  the  same 
angle  of  visual  field  as  the  object  it  represents  in  the  real  world  and  would  also  impart 
correct  motion  parallax  and  motion  perspective  cues  in  relation  its  virtual 
surroundings  as  the  viewer  changes  position. 
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-  Differences  between  human  eye  and  virtual  camera  image  geometry  are  discussed  with 
reference  to  view  representation,  depth  of  field  representation,  image  separation  for 
stereoscopic  fusion,  window  effect  in  stereoscopic  image  fusion,  foveal  image  display, 
and  correct  focal  length  and  viewport  specification. 

Differences  between  human  eye  and  virtual  camera  image  QUALITY  such  as 
luminance,  resolution,  color,  and  texture  are  discussed  as  well  as  differences  between 
human  visual  system  and  virtual  camera  image  DYNAMICS  such  as  frame  rate  and 
image  generation  lag,  axis  of  rotation  for  changing  viewpoint  (tracker  offset),  image 
and  spatial  distortions  from  non-parallel  optical  axes,  and  infinity  optics  versus  variable 
accomodation  performance. 

Multisensory  virtual  representations  provide  an  additional  dimension  to  the 
telepresence  experience  and  are  discussed  in  relation  to  synchronization  with  visual 
imagery,  auditory  localization  and  3D  sound  cues,  tactile  feedback,  and  visceral 
simulation  with  motion  platforms. 
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VIRTUAL  ACOUSTIC  DISPLAYS 


Elizabeth  M.  Wenzel 
NASA  Ames  Research  Center 
Moffett  Field,  California 


As  with  most  research  in  information  displays,  virtual  displays  have  generally  emphasized  visual 
information.  Many  investigators,  however,  have  pointed  out  the  importance  of  the  auditory  system  as 
an  alternative  or  supplementary  information  channel  (e.g.,  Deatherage,  1972;  Doll,  et.  al.,  1986; 
Patterson,  1982;  Caver,  1986).  A  three-dimensional  auc-^ory  display  can  potentially  enhance  infor¬ 
mation  transfer  by  combining  directional  and  iconic  information  in  a  quite  naturalistic  representation 
of  dynamic  objects  in  the  interface.  Borrowing  a  term  from  Gaver  (1986),  an  obvious  aspect  of 
“everyday  listening”  is  the  fact  that  we  live  and  listen  in  a  three-dimensional  world.  Indeed,  a  pri¬ 
mary  advantage  of  the  auditory  system  is  that  it  allows  us  to  monitor  and  identify  sources  of  infor¬ 
mation  from  all  possible  locations,  not  just  the  direction  of  gaze.  This  feature  would  be  especially 
useful  in  an  application  that  is  inherendy  spatial,  such  as  an  air  traffic  control  display  for  the  tower  or 
cockpit.  A  further  advantage  of  the  binaural  system,  often  referred  to  as  the  “cocktail  party  effect” 
(Cherry,  1953),  is  that  it  improves  the  intelligibility  of  sources  in  noise  and  assists  in  the  segregation 
of  multiple  sound  sources.  This  effect  could  be  critical  in  applications  involving  encoded  nonspeech 
messages  as  in  scientific  “visualization,”  the  acoustic  representation  of  multi-dimensional  data  (e.g., 
Bly,  1982)  and  the  development  of  alternative  interfaces  for  the  visually-impaired  (Edwards,  1989; 
Loomis,  et.  al.,  1990).  Another  aspect  of  auditory  spatial  cues  is  that,  in  conjunction  with  other 
modalities,  it  can  act  as  a  potendator  of  information  in  the  display.  For  example,  visual  and  auditory 
cues  together  can  reinforce  the  information  content  of  the  display  and  provide  a  greater  sense  of 
presence  or  realism  in  a  manner  not  readily  achievable  by  either  modality  alone  (Colquhoun,  1975; 
Warren,  et.  al.,  1981;  O’Leary  and  Rhodes,  1984).  This  phenomenon  will  be  particularly  useful  in 
telepresence  applications,  such  as  advanced  teleconferencing  environments,  shared  electronic 
workspaces,  and  monitoring  telerobotic  activities  in  remote  or  hazardous  situations.  Thus,  the  com¬ 
bination  of  direct  spatial  cues  with  good  principles  of  iconic  design  could  provide  an  extremely  pow¬ 
erful  and  information-rich  display  which  is  also  quite  easy  to  use. 

This  type  of  display  could  be  realized  with  an  array  of  real  sound  sources  or  loudspeakers  (Doll, 
et.  al.,  1986;  Calhoun,  et.  al,  1987).  An  alternative  approach,  recently  developed  at  NASA- Ames, 
generates  externalized,  three-dimensional  sound  cues  over  headphones  in  realtime  using  digital  sig¬ 
nal-processing  (Wenzel,  et.  al.,  1988a).  Here,  the  synthesis  technique  involves  the  digital  generation 
of  stimuli  using  Head-Related  Transfer  Functions  (HRTFs)  measured  in  the  two  ear-canals  of  indi¬ 
vidual  subjects  (see  Wightman  and  Kistler,  1989a:.  Up  to  four  moving  or  static  sources  can  be  simu¬ 
lated  in  a  head-stable  environment  by  digital  filtering  of  arbitrary  signals  with  the  appropriate 
HRTFs.  This  type  of  presentation  system  is  desirable  because  it  allows  complete  control  over  the 
acoustic  waveforms  delivered  to  the  two  ears  and  the  ability  to  interact  dynamically  with  the  virtual 
display.  Other  similar  approaches  include  an  analog  system  developed  by  Loomis,  et.  al.  (1990)  and 
digital  systems  which  make  use  of  transforms  derived  from  normative  mannikins  and  simulations  of 
room  acoustics  (Genuit,  1986;  Posselt,  et.  al.,  1986;  Persterer,  1989;  Lehnert  and  Blauert,  1989). 
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Such  an  interface  also  requires  the  ''areful  psychophysical  evaluation  of  listeners’  ability  to  accu¬ 
rately  localize  the  virtual  or  synthetic  sound  sources.  For  example,  a  recent  study  by  Wightman  and 
Kistler  (1988b)  confirmed  the  perceptual  adequacy  of  the  basic  technique  for  static  sources;  source 
azimuth  was  synthesized  nearly  perfectly  for  all  listeners  while  source  elevation  was  somewhat  less 
well-defined  in  the  headphone  conditions. 

From  an  applied  standpoint,  measurement  of  each  potential  listener’s  HRTFs  may  not  be  possi¬ 
ble  in  practice.  It  may  also  be  the  case  that  the  user  of  such  a  display  will  not  have  the  opportunity 
for  extensive  training.  Thus,  a  critical  research  issue  for  virtual  acoustic  displays  is  the  degree  to 
which  the  general  population  of  listeners  can  obtain  adequate  localization  cues  from  stimuli  based  on 
non-individualized  transforms.  Preliminary  data  (Wenzel,  et.  al.,  1988b)  suggest  that  using  non-lis¬ 
tener-specific  transforms  to  achieve  synthesis  of  localized  cues  is  at  least  feasible. 

For  experienced  listeners,  localization  performance  was  only  slightly  degraded  compared  to  a 
subject’s  inherent  ability,  even  for  the  less  robust  elevation  cues,  as  long  as  the  transforms  were 
derived  from  what  one  might  call  a  “good”  localizer.  Further,  the  fact  that  individual  differences  in 
performance,  particularly  for  elevation,  could  be  traced  to  acoustical  idiosyncracies  in  the  stimulus 
suggests  that  it  may  eventually  be  possible  to  create  a  set  or  “universal  transforms”  by  appropriate 
averaging  (Genuit,  1986)  and  data  reduction  techniques  (e.g.,  principal  components  analysis),  or 
perhaps  even  enhancing  the  spectra  of  empirically-derived  transfer  functions  (Durlach  and  Pang, 
1986). 

Alternatively,  even  inexperienced  listeners  may  be  able  to  adapt  to  a  particular  set  of  HRTFs  as 
long  as  they  provide  adequate  cues  for  localization.  A  reasonable  approach  is  to  use  the  HRTFs  from 
a  subject  whose  measurements  have  been  “behaviorally-calibrated”  and  are  thus  correlated  with 
known  perceptual  ability  in  both  free-field  and  headphone  conditions.  In  a  recently  completed  study, 
sixteen  inexperienced  listeners  judged  the  apparent  spatial  location  of  sources  presented  over  loud¬ 
speakers  in  the  free-field  or  over  headphones.  The  headphone  stimuli  were  generated  digitally  using 
HRTFs  measured  in  the  ear  canals  of  a  representative  subject  (a  “good  localizer”)  from  Wightman 
and  Kistler  (1988a,b).  For  twelve  of  the  subjects,  localization  performance  was  quite  good,  with 
judgements  for  the  non-individualized  stimuli  nearly  identical  to  those  in  the  free-field. 

In  general,  these  data  suggest  that  most  listeners  can  obtain  usefu*  directional  information  from 
an  auditory  display  without  requiring  the  use  of  individually-tailored  HRTFs.  However,  a  caveat  is 
important  here.  The  results  described  above  are  based  on  analyses  in  which  errors  due  to  front/back 
confusions  were  resolved.  For  free-field  versus  simulated  free-field  stimuli,  experienced  listeners 
exhibit  front/back  confusion  rates  of  about  5  vs.  10  %  and  inexperienced  listeners  show  average  rates 
of  about  22  vs.  34  %.  Although  the  reason  for  such  confusions  is  not  completely  understood,  they  are 
probably  due  in  large  part  to  the  static  nature  of  the  stimulus  and  the  ambiguity  resulting  from  the  so- 
called  cone  of  confusion  (see  Blauert,  1983).  Several  stimulus  characteristics  may  help  to  minimize 
these  errors.  For  example,  the  addition  of  dynamic  cues  correlated  with  head-motion  and  well- 
controlled  environmental  cues  derived  from  models  of  room  acoustics  may  improve  the  ability  to 
resolve  these  ambiguities. 
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Natural  environments  have  a  content,  i.e.,  the  objects  in  them;  a  geometry,  i.e.,  a  pattern  of  rules 
for  positioning  and  displacing  the  objects;  and  a  dynamics,  i.e.,  a  system  of  rules  describing  the 
effects  of  forces  acting  on  the  objects.  Human  interaction  with  most  common  natural  environments 
has  been  optimized  by  centuries  of  evoiution.  Virtual  environments  created  through  the  human-com¬ 
puter  interface  similarly  have  a  content,  geometry,  and  dynamics,  but  the  arbitrary  character  of  the 
computer  simulation  creating  them  does  not  in  'ure  that  human  interaction  with  these  virtual  envi¬ 
ronments  will  be  natural.  The  interaction,  indeed,  could  be  supernatural  but  it  also  could  be 
impossible. 

An  important  determinant  of  the  comprehensibu.’ty  of  a  virtual  environment  is  the  correspon¬ 
dence  between  the  environmental  frames  of  reference  and  those  associated  with  the  control  of  envi¬ 
ronmental  objects.  The  effects  of  rotation  and  displacement  of  control  frames  of  reference  w  ith 
respect  to  corresponding  environmental  references  differ  depending  upon  whether  perceptual  judge¬ 
ment  or  manual  tracking  pert'ormancp  is  measured. 

The  perceptual  effects  of  frame  of  reference  displacement  may  be  analyzed  in  terms  of  distor¬ 
tions  in  ♦he  process  or  virtualizing  the  synthetic  environmental  space.  The  effects  of  Fame  of  refer¬ 
ence  displacement  and  rotation  have  been  studie''  by  asking  subjects  to  estimate  exocentric  direction 
in  a  virtual  space.  Exocentric  judgements  involve  the  estimation  of  ♦he  orientation  of  one  external 
object  with  respect  to  an  external  frame  of  reference.  They  may  be  contrasted  with  egocentric 
judgements  in  which  objects  r  rientation  is  judged  with  respect  to  the  viewer  himself.  Though 
alternative  models  of  the  er-nrs  of  exocentric  direction  may  be  compared,  the  most  parsimonious 
explanation  may  be  that  viewers  misjudge  the  direction  of  the  v  iew  vector  used  to  produce  the 
display.  The  motor  effects  of  frame  of  reference  misalignment  have  been  studied  using  a  technique 
of  three  dimensional  tracking  in  which  a  target  moves  irregularly  in  3  axes.  Pursuit  tracking  errors 
may  be  separated  into  ''nearly  additive  visual  and  visual-motor  components,  but  the  pattern  of  error 
as  a  function  of  angular  misalignment  may  not  generalize  to  compensatory  t'-acking  or  to  non- 
holonomic  control  modes. 
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THE  EXPLORATION  METAPHOR 


Michael  W.  McGreevy 
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NASA’s  experience  in  planetary  exploration  has  demonstrated  that  the  desktop  workstation  is 
inadequate  for  many  visualization  situations.  The  primary  mission  displays  for  the  unmanned  Sur¬ 
veyor  missions  to  the  moon  during  the  mid-1960s,  for  example,  were  environmental  images  assem¬ 
bled  on  the  inside  surfaces  of  spherical  shells.  Individual  images  taken  in  1971  by  the  first  Mars 
orbiter.  Mariner  9,  were  applied  to  the  outside  of  four-foot  diameter  spheres,  creating  composite  pho¬ 
tographs  of  the  entire  planet.  During  the  Viking  mission  to  Mars  in  1976,  initial  orbiter  photos  were 
mosaicked  on  large  tables  or  on  walls  to  serve  as  group  decision-making  displays.  The  environments 
surrounding  the  two  Viking  landers  were  visualized  using  s>ereo  and  panoramic  images,  desktop 
models,  and  full-sized  replicas  of  the  lander  and  the  surrounding  rocks. 

Future  exploration  missions  will  greatly  benefit  from  advances  in  digital  computer  and  display 
technology,  but  there  remain  unmet  user  interface  needs.  Unfortunately,  the  desktop  workstation  is 
generally  considered  to  be  the  starting  point  of  user-computer  interface  design,  and  most  current 
human-computer  interface  research  emphasizes  the  desktop  metaphor.  In  this  view,  computing  is 
thought  to  be  comparable  to  paperwork,  and  the  objects  and  functions  available  to  the  user  reflect 
this  emphasis.  Point-and-click  interaction  with  icons  and  windows  is  supportive  of  operations  per¬ 
formed  on  text  and  simple  pictures,  but  this  approach  is  inadequate  for  planetary  exploration. 

Alternative  user  interfaces  and  metaphors  are  needed  for  planetary  exploration  and  other  interac¬ 
tions  with  complex  spatial  environments.  These  interfaces  and  metaphors  would  enable  the  user  to 
directly  explore  environments  and  naturally  manipulate  objects  in  those  environments.  Personal 
simulators,  virtual  workstations,  and  telepresence  user  interfaces  are  systems  capable  of  providing 
this  integration  of  user  space  and  task  space.  The  Exploration  Metaphor  is  a  useful  concept  for 
guiding  the  design  of  user  interfaces  for  virtual  environments  and  telepresence.  To  apply  the  Explo¬ 
ration  Metaphor  is  to  assert  that  computing  is  like  exploration,  and  to  support  objects,  operations, 
and  contexts  comparable  to  those  encountered  in  the  exploration  of  natural  environments.  The 
Exploration  Metaphor,  under  development  for  user  interfaces  in  support  of  NASA’s  planetary  explo¬ 
ration  missions  and  goals,  will  also  benefit  other  applications  where  complex  spatial  information 
must  be  visualized. 

Visualization  methods  and  systems  for  planetary  exploration  are  becoming  increasingly  inte¬ 
grated  and  interactive  as  computing  technology  improves.  These  advances  will  benefit  from  virtual 
environment  and  telepresence  interface  technology.  A  key  development  has  been  the  processing  of 
multiple  images  and  other  sensor  data  to  create  detailed  digital  models  of  the  planets  and  moons. 

Data  from  images  of  the  Earth,  Mars,  and  Miranda  (a  moon  of  Uranus),  for  example,  have  been  con¬ 
verted  into  three  dimensional  models,  and  dynamic  virtual  fly-overs  have  been  computed  as  demon¬ 
strations  .Similar  processing  of  lower  altitude  plmtography  and  the  u.se  of  computer  aided  design 
tool  promise  to  produce  very  detailed  models  in  the  future.  Currently,  the  Magellan  spacecraft  is 
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more  than  half  way  to  Venus  on  a  mission  to  digitize  the  entire  surface  of  that  planet,  and  this,  too, 
will  provide  an  interactive  digital  model  for  virtual  exploration  on  Earth. 

Future  plans  call  for  multiple  unmanned  rovers  to  traverse  the  surface  of  Mars,  enabling  geosci¬ 
entists  located  at  a  central  Mars  base  to  explore  the  planet  via  telepresence.  In  addition.  Earth-bound 
mission  operators  and  scientists  will  be  able  to  access  integrated  digital  models  of  Mars  as  the  mis¬ 
sions  progress.  The  Exploration  Metaphor  will  contribute  to  the  design  of  user  interfaces  for  these 
applications. 
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Early  studies  in  our  laboratory  showed  that  cats  could  be  trained  to  alter  spontaneous  EEG  pat¬ 
terns  through  contingent  food  reinforcement  of  operant  configurations.  In  particular,  rhythmic  EEG 
patterns  in  sensorimotor  cortex,  which  were  related  behaviorally  to  motionlessness,  were  facilitated 
in  this  manner,  and  showed  sustained  alteration  across  states  and  time.  Electrophysiological  changes 
in  the  sensorimotor  pathways  were  found  to  accompany  this  effect.  Additionally,  several  striking 
behavioral  changes  were  seen  as  well,  including  in  particular  an  enhancement  of  sleep  and  an  eleva¬ 
tion  of  seizure  threshold  to  epileptogenic  agen:  j. 

This  latter  observation  raised  the  possibility  that  human  seizure  disorders  might  be  influenced 
therapeutically  by  similar  training.  However,  in  man  the  use  of  food  reinforcement  was  not  practical. 
Further,  unlike  our  cats,  human  epileptics  present  with  characteristic  EEG  abnormalities.  An  evalua¬ 
tion  of  these  abnormalities  from  the  perspective  of  our  cat  data  showed  that  while  normal  humans 
had  analogous  patterns,  untreated  epileptics  were  often  deficient  in  normal  rhythmic  sensorimotor 
activity  and  had  EEG  patterns  dominated,  instead,  by  slower  and  more  poorly  organized  patterns.  It 
thi's  became  necessary  to  develop  a  very  different  strategy  in  applying  this  methodology  to 
epileptics. 

Our  objective  in  human  EEG  feedback  training  became  not  only  the  facilitation  of  normal 
rhythmic  patterns,  but  also  the  suppression  of  abnormal  activity,  thus  requiring  complex  contingen¬ 
cies  directed  to  the  noimalization  of  the  sensorimotor  EEG.  To  achieve  this,  a  multicomponent  fre¬ 
quency  analysis  was  developed  to  extract  and  separate  normal  and  abnormal  elements  of  the  EEG 
signal.  Each  of  these  elements  was  transduced  to  a  specific  component  of  a  visual  display  system, 
and  the.se  were  combined  through  logic  circuits  to  present  the  subject  with  a  symbolic  display.  This 
display  allowed  the  subject  to  score  “points”  in  a  variety  of  game  formats  through  the  normalization 
of  EEG  characteristics.  Variable  criteria  provided  for  the  gradual  shaping  of  EE<' i  elements  towards 
the  desired  normal  pattern.  Some  50-70%  of  patients  with  poorly  controlled  seizure  disorders  experi¬ 
enced  therapeutic  benefits  from  this  approach  in  our  laboratory,  and  subsequently  in  many  other  lab¬ 
oratories  as  Well. 

A  more  recent  application  of  this  approach  to  the  modification  of  human  Drain  function  in  our 
laboratory  has  been  directed  to  the  dichotomous  problems  of  task  overload  and  underload  in  the 
contemporary  aviation  environment.  At  least  70%  of  all  aviation  accidents  have  been  attributed  to 
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the  impact  of  these  kinds  of  problems  on  crew  performance.  The  use  of  the  EEG  in  this  context  has 
required  many  technical  innovations  and  the  application  of  the  latest  advances  in  EEG  signal  analy¬ 
sis.  Our  first  goal  has  been  the  identification  of  relevant  EEG  characteristics.  Additionally,  we  have 
developed  a  portable  recording  and  analysis  system  for  application  in  this  context.  Findings  from 
laboratory  and  in-flight  studies  suggest  that  we  will,  indeed,  be  able  to  detect  appropriate  changes  in 
brain  function,  and  feed  this  information  to  on-board  computers  for  modification  of  mission 
requirements  and/or  crew  status. 

Supported  by  the  Veterans  Administration,  the  Public  Health  Service,  NASA,  and  Northrop 
Aviation. 
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THE  “TELEPRASENZ”  CONSORTIUM:  STRUCTURE  &  INTENTIONS 


Jens  Blauert 

Ruhr-Universitat  Bochum 
Bochum,  FR  Germany 


The  Teleprasenz-Consortium  is  an  open  group  of  currently  37  scientists  of  different  disciplines 
who  devote  a  major  part  of  their  research  activities  to  the  foundations  of  telepresence  technology. 
The  necessary  support  for  their  work  is  provided  by  German  federal  and  state  sources  as  well  as  by 
European  agencies  and  industry.  The  activities  of  the  consortium  are  organized  into  three  main 
branches;  virtual  environment,  surveillance  and  control  systems  and  speech  and  language  technol¬ 
ogy.  In  the  following,  a  brief  summary  of  the  main  activities  in  these  areas  is  given. 

Telepresence  technology  -  in  this  context  -  is  basically  understood  as  a  means  to  bridge  spatial 
and  temporal  gaps  as  well  as  certain  kinds  of  concealment,  inaccessibility  and  danger  of  exposure 
(e.g.,  microscopic  dimensions,  non-perceptibility  by  human  senses,  sensoric  disabilities,  speech  and 
language  problems,  load  of  emissions). 

Virtual  Environment:  In  this  group,  it  is  the  aim  to  provide  a  virtual  environment  for  specific 
tasks  such  as  telesurveillance  or  telecooperation  applications.  Although  the  main  focus  was  directed 
on  the  auditoiy  and  visual  components  so  far,  future  work  will  include  tactile  and  kinesthetic  cues. 

Surveillance  and  Control  Systons;  This  group  is  mainly  engaged  in  tasks  wliich  concern 
control  technology,  diagnostics  of  complex  systems  in  biology  and  technology  as  well  as  in 
accompanying  problems  in  signal  processing  and  software  technology. 

Speech  and  Language  Technology:  This  group  is  concerned  with  several  projects  in  the  areas 
of  automatic  speech  in-  and  output,  speech  coding  and  symbolic  processing  fore  telepresence 
scenarios. 

Current  research  proposals  of  the  Teleprasenz  Consortium  include  a  telemanipulator  system  and 
a  testbed  for  polysensory  virtual  environment  as  well  as  many  smaller  research  and  development 
contracts.  An  information  brochure  on  Teleprasenz  (in  German)  will  be  available  at  the  conference. 
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MACfflNE  PERCEPTION  AND  ITS  ROLE  IN  VIRTUAL  ENVIRONMENTS 

S.  K.  Ganapathy 
AT&T  Bell  Laboratories 
Holmdel,  New  Jersey 

(No  written  material  provided) 
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SUMMARIES  OF  PANEL  DISCUSSIONS 


PANEL  DISCUSSION  ON  TELEPRESENCE  AND  SENSORIMOTOR 

ADAPTATION 


R.  B.  Welch,  Chair 
NASA  Ames  Research  Center 

J.  Loomis 

University  of  California,  Santa  Barbara 
R.  Pepper 

Science  Applications  International,  Inc. 
M.  Kreuger 

Artificial  Reality  Corporation 
and 

T.  Etter 
ANPA  West 


Four  questions  were  addressed.  The  following  represents  a  synopsis  of  the  thoughts  of  each 
seminar  member  (or  members)  with  respect  to  his/her  assigned  item. 

1.  What  do  we  mean  by  the  term  “telepresence”  and  how  can  we  measure  it?  (Jack  Loomis) 

If  virtual  and  remotely  sensed  environments  are  comprised  of  the  same  sensory  cues  as  near- 
sensed  worlds  (i.e.,  “equivalent  configurations”),  telepresence  will  be  complete.  This  rather  self-evi¬ 
dent  observation  leads  to  the  much  more  interesting  issue  of  why  some  everyday  perceptual  events 
are  externalized  (experienced  as  non-self),  while  others  are  not.  Pain,  for  example,  is  perceived  as 
internal,  while  most  objects  in  our  perceptual  experience  are  externalized. 

It  has  been  proposed  that  the  degree  of  experienced  extemalization  is  related  to  the  range  of 
variation  in  the  efference  to  the  motor  system  that  results  in  some  invariance  in  the  sensory  inflow 
(e.g.,  Brunswik,  Gibson,  Polanyi).  According  to  this  view,  pain  is  completely  internalized  because 
the  sensory  signals  are  constant  with  changes  in  motor  outflow.  A  positive  afterimage  produces 
intermediate  extemalization  because  the  change  in  its  angular  position  within  the  visual  field  is 
strictly  correlated  with  eye  movements.  The  most  compelling  sense  of  extemalization  arises  when 
sensory  input  from  all  of  the  senses  provides  converging  evidence,  under  active  exploration,  of  some 
fixed  entity.  If  this  compelling  extemalization  arises  when  using  a  virtual  display  or  teleoperator,  we 
say  that  there  is  “telepresence.”  but  just  as  interesting  is  the  extemalization  that  occurs  with  ordinary 
near-sensing  of  the  world. 

Finally,  a  necessary  condition  for  extemalization  (telepresence)  when  using  a  virtual  display  or 
teleoperator  is  that  the  observer  be  capable  of  internalizing  the  dynamics  of  the  linkage  (plant) 
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mediating  contact  with  the  virtual  or  remote  environment.  If  the  linkage  is  too  unnatural,  arbitrary,  or 
non-linear  for  this  to  occur,  the  observer  will  experience  little  extemalization  of  the  distal  environ¬ 
ment.  Nevertheless,  adaptation  may  serve  to  re-establish  some  sense  of  extemalization.  Once  this 
happens,  the  plant  model  becomes  transparent  to  the  observer  and  extemalization  of  the  distal  envi¬ 
ronment  results.  A  common  example  involves  long-cane  training  for  the  blind.  At  first,  the  trainee 
fumbles  a  lot  and  is  very  aware  of  the  cane  rather  than  the  environment.  As  skill  develops,  however, 
manipulation  of  the  cane  becomes  more  automatic,  resulting  eventually  in  its  transparency  and  in  the 
extemalization  of  the  environment  with  which  the  tip  of  the  cane  is  making  contact. 

2.  Of  what  good  is  telepresence?  Does  it  contribute  to  performance?  Is  it  self-evident  that  the 
more  telepresence,  the  better?  (Ross  Pepper) 

A  number  of  benefits  result  when  the  experience  of  telepresence  is  increased.  First,  under  height¬ 
ened  telepresence,  less  operator  training  is  necessary.  Second,  teleoperator  performance  under  these 
conditions  is  more  comparable  to  direct  human  performance.  Third,  optimal  telepresence  can  be 
expected  to  increase  the  “emotionality”  of  the  remote  environment,  depending  on  the  content  of  the 
replicated  sensory  experience.  An  example  of  the  latter  point  is  the  enjoyable  visceral  feelings  of 
actually  “being  there”  experienced  by  the  viewers  of  a  very  realistic,  action-filled  movie.  Finally,  the 
very  search  for  telepresence  will  stimulate  the  conceptual  analysis  of  perceptual  issues  (see  Loomis’ 
remarks). 

Telepresence  contributes  to  performance,  but  not  without  certain  costs.  The  benefits  of  telepres¬ 
ence  to  performance  are  that  it  (1)  reduces  errors  and  task  performance  variability,  (2)  increases  pre¬ 
cision.  (3)  reduces  training  costs,  (4)  enables  performance  similar  to  direct  viewing  conditions,  and 
(5)  results  in  greater  adaptability  to  changing  task  conditions. 

The  costs  of  enhanced  telepresence  are  (1)  an  increase  in  system  complexity,  (2)  reduced  relia¬ 
bility,  (3)  increased  initial  investment,  and  (4)  increased  maintenance  costs. 

It  may  be  concluded  that  teleoperator  system  enhancements  that  produce  marked  increases  in 
performance  generally  result  in  a  heightened  experience  of  telepresence.  Examples  of  such 
enhancements  are  changing  a  monocular  visual  display  to  a  stereoscopic  display  and  providing  force 
feedback  for  the  manipulating  hand. 

3.  Which  characteristics  of  the  teleoperator  or  virtual  environment  system  contribute  to 
telepresence,  which  are  irrelevant,  and  which  degrade  it?  (Myron  Krueger  and  Tom  Etter) 

(Krueger’s  remarks) 

A  number  of  factors  influencing  telepresence  or  artificial  worlds  can  be  addressed,  the  most 
inlluential  of  which  is  maintaining  a  short  delay  between  action  and  response.  This  is  more  important 
than  a  high  sample  rate  by  means  of  pipelining  techniques.  It  is  also  claimed  that  short  delays  are 
more  important  than  a  realistic  portrayal  in  graphic  worlds.  The  allowable  delay  varies  with  the  task. 
In  particular,  fine  manipulation  requires  shorter  delays  than  docs  gross  manipulation.  The  ability  to 
look  around  suffers  with  delays  longer  than  .03  seconds;  it  can  be  done,  but  the  operator  is  aware  of 
the  lag.  In  general,  there  is  a  decline  in  possible  telepresence  with  distance.  Beyond  a  rather  short 
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distance,  the  only  way  the  immediacy  of  telepresence  can  be  attained  is  by  interacting  locally  with  a 
graphic  world. 

Another  dimension  that  may  be  explored  is  the  effect  of  unencumbering  sensors  and  actuators 
designed  to  increase  telepresence.  The  first  observation  is  that  the  presence  of  encumberance  signals 
a  distance  from  the  world  in  which  one  wishes  to  feel  present.  Also,  the  discomfort  caused  by  pro¬ 
longed  encumberance  can  lead  to  a  net  loss  in  telepresence. 

It  should  also  be  observed  that  realism  may  not  necessarily  be  the  ultimate  goal  of  telepresence. 
Natural  behavior  can  be  instantly  achieved  in  2-D  realities  which  are  easier  to  enter  and  exit  than  are 
3-D  worlds  than  can  only  be  experienced  by  “suiting  up.”  Furthermore,  if  artificial  worlds  are  to  be 
used  for  applied  purposes,  many  traditional  techniques  will  be  demanded  that  violate  realism.  While 
people  may  initially  obey  the  spatial  metaphor  to  go  from  one  point  in  a  virtual  world  to  another, 
later  they  will  probably  want  to  make  the  trip  instantly  by  means  of  virtual  teleportation. 

A  final  event  that  will  affect  telepresence  in  artificial  realities,  as  well  as  in  tasks  where  two 
telerobots  cooperate  at  a  remote  site,  is  the  representation  of  the  operator  at  the  effector  end.  If  the 
individual  is  to  be  represented,  it  is  inevitable  that  a  face  must  be  provided  and  no  method  of  three- 
dimensional  facial  image  capture  and  portrayal  currently  exists. 

(Etter’s  remarks) 

What  enhances  telepresence?  Here’s  a  related  question:  What  enhances  the  “realism”  of  artificial 
food?  In  this  case  we’re  clearly  dealing  with  two  separate  issues,  taste  and  food  value.  I  suggest  that 
essentially  the  same  two  issues  arise  in  the  case  of  “artificial  presence.”  We  must  distinguish  the 
sense  ot  presence  trom  the  physical  cuniponent  of  presence,  whatever  exactly  that  may  be.  Suppose 
that  you  are  wired  into  a  computer-generated  virtual  environment.  Does  the  computer  supply  this 
physical  component  of  presence?  Do  the  wires  carry  it? 

Bell’s  theorem  shows  that  any  informational  model  of  quantum  phenomena  (e.g.,  computer 
simulation)  must  contain  faster-than  light  information  paths,  thereby  contradicting  the  theory  of 
relativity.  Thus,  if  relativity  does  in  fact  govern  the  transmission  of  information,  the  connections 
among  things  in  the  world  cannot  be  purely  informational — they  must  also  transmit  something  else. 
Let’s  call  this  something  else  the  L  factor. 

The  L  factor  involves  negative  probabilities  which  makes  its  meaning  quite  mysterious.  How¬ 
ever.  its  mathematics  are  very  simple  and  it  leads  to  an  elegant  and  robust  explanation  of  basic 
quantum  theory.  The  only  alternative  to  the  L  factor  is  to  assume  the  existence  of  hidden,  non-local 
information  paths.  This  is  easy  to  interpret  in  familiar  ways,  but  it  fills  the  world  with  unobservable 
entities  and  generally  creates  a  mess. 

Assuming  that  we  go  with  the  L  factor,  what  might  this  mean  for  telepresence?  Could  the  L  fac¬ 
tor  be  the  “food  value”  of  physical  presence?  Could  there  be  a  long-term  danger  in  going  without  it, 
like  eating  food  with  vitamins?  With  a  more  scientifically  advanced  technology,  might  we  be  able  to 
build  virtual  environments  that  supply  it? 
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4.  To  what  extent  is  it  necessary  for  the  human  operator  to  adapt  to  the  teleoperation  or  vir¬ 
tual  environment  situation  in  order  to  experience  a  strong  sense  of  telepresence  and/or  to 
function  effectively?  (Robert  Welch) 

With  current  technology,  remotely  sensed  and  virtual  worlds  inevitably  entail  significant  inter- 
sensory  and  sensorimotor  discordances,  which  serve  to  degrade  both  performance  and  the  experience 
of  telepresence.  However,  it  is  likely  that  these  problems  can  be  overcome,  at  least  partially,  by 
means  of  adaptation  or  adaptation-like  processes,  of  which  there  are  four  varieties;  (1)  intersensory 
bias,  (2)  perceptual  adaptation,  (3)  perceptual-motor  adaptation,  and  (4)  habituation. 

The  most  commonly  cited  example  of  intersensory  bias  is  “visual  capture” — the  “dominance”  of 
vision  over  felt  limb  position.  It  represents  an  immediate  and  very  short-lived  resolution  of  the  inter¬ 
sensory  conflict  since  it  disappears  as  soon  as  vision  is  precluded.  The  magnitude  of  intersensory 
bias  is  inversely  related  to  the  size  of  the  intersensory  discrepancy  and  it  is  facilitated  by  familiarity 
with  the  intersensory  relationship  (e.g.,  a  view  of  the  hand)  as  well  as  by  the  presence  of  congruent 
intersensory  characteristics  of  the  stimulus  object  such  as  its  felt  and  seen  shape,  motion,  orientation, 
and  size.  These  same  factors  are  presumably  involved  in  creating  a  compelling  experience  of 
telepresence. 

Perceptual  adaptation  represents  a  compensatory  recalibration  of  one  or  both  of  the  sensory  sys¬ 
tems  that  serves  to  resolve  the  intersensory  discrepancy.  In  contrast  to  intersensory  bias,  it  persists  as 
an  aftereffect.  This  type  of  adaptation  will  not  occur  with  very  large  discrepancies,  such  as 
1 80-degree  optical  rotation. 

Perceptual-motor  adaptation  refers  to  changes  in  the  observer’s  performance.  If  perceptual  adap¬ 
tation  has  occurred,  perceptual-motor  adaptation  is  likely  to  follow  suit,  whereas  the  reverse  does  not 
always  hold,  particularly  with  very  large  rearrangements.  For  example,  even  though  after  several 
days  of  practice  one  will  get  quite  good  at  operating  in  a  1 80-degree  rotated  visual  world,  it  will 
continue  to  look  upside  down. 

Factors  tliat  affect  the  magnitude  of  perceptual  and/or  perceptual-motor  adaptation  are  active 
(versus  passive)  movement,  immediate  feedback,  error  correction,  distributed  practice,  and  incre¬ 
mental  exposure.  Presumably,  these  variables  would  similarly  influence  adaptation  to  rearranged 
remote  or  virtual  environments. 

After  prolonged  experience  with  the  rearranged  sensory  environment,  it  will  come  to  look  famil¬ 
iar.  a  phenomenon  referred  to  here  as  habituation.  This  may  occur  even  when  no  perceptual  adapta¬ 
tion  has  taken  place.  For  example,  well  practices  wearers  of  inverting  goggles  become  so  accus¬ 
tomed  to  their  upside-down  visual  world  that  for  periods  of  time  they  pay  no  attention  to  its  orienta¬ 
tion.  It  is  likely  that  at  those  times  the  feeling  of  presence  is  strong  and  that  after  much  practice  the 
same  would  be  true  for  a  remote  or  virtual  world  that  may  have  initially  appeared  quite  discrepant  or 
bizarre. 

Finally,  if  and  when  it  is  possible  to  construct  a  teleoperator  or  virtual  environment  system  that  is 
completely  accurate  in  its  portrayal  of  the  naturally  sensed  world,  there  will  be  little  to  which  the 
operator  will  have  to  adapt.  However,  even  with  such  an  ideal  system,  some  adaptation  will  be 
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necessary  to  overcome  the  inter-operator  differences  in  perccp*^ion  and  reponse  that  have  been 
V'idely  reporteo. 
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PANEL  DISCUSSION  ON  PERFORMANCE  MEASUREMENT  AND 

EVALUATION  METHODS 


Daniel  T.  Ling,  Chair  (IBM),  Ruzena  Bajcsy  (University  of  Pennsylvania),  Anatal  Bejczy  (Johnson 
Propulsion  Laboratory),  Beth  Marcus  (Exos,  Inc.),  and  Misha  Pavel  (Stanford  University) 


The  seminar  was  introduced  by  Daniel  Ling  who  presented  a  brief  outline  of  a  number  of  key 
issues  which  need  to  be  addressed  within  the  context  of  performance  measurement  and  evaluation 
methods. 

1 .  The  context  of  any  evaluation  must  be  the  task.  How  should  the  tasks  be  defined  and 
decomposed? 

2.  What  needs  to  be  measured?  This  really  means  obtaining  the  data  needed  to  develop  models. 
Models  are  needed  for  both  the  human  operator  as  well  as  the  hardware/software  system. 

3.  What  criteria  should  be  used  for  the  measurements?  This  might  include  task  performance  (mean 
and  variance),  training  required,  ease  of  recall,  “likeability”  which  may  NOT  correspond  to  any 
simple  measure  of  performance,  etc. 

4.  Need  for  rigorous  experiments  as  well  as  general  observations  and  rules  of  thumb. 

Misha  Pavel  advocated  evaluations  based  upon  explicitly  specified  system  models.  He  also  dis¬ 
cussed  the  importance  of  task  analysis. 

Model  Based  Evaluation  (Misha  Pavel) 

Evaluation  of  interactive  human-machine  systems  is  usually  very  difficult  because  there  are  too 
many  variables,  too  many  conditions  and  the  results  obtained  with  one  system  are  often  difficult  to 
apply  to  other  similar  ones  (generalization  problem).  Frequently,  the  only  way  to  make  this  problem 
manageable  is  to  formulate  an  explicit  “computational”  model  of  all  the  components  of  the  system 
including  the  human  operator.  Such  a  model,  because  of  the  constraints  it  imposes,  will  permit  con¬ 
siderably  more  efficient  evaluation.  For  example,  many  formal  models  specify  a  functional  relation¬ 
ship  among  variables  providing  only  a  few  free  parameters.  Then,  instead  of  evaluating  every  possi¬ 
ble  combination  of  all  the  variables,  one  must  only  determine  a  smaller  set  of  parameters. 

Whenever  performance  of  a  system  is  measured,  there  is  an  underlying  model  even  if  it  is  not 
explicitly  specified.  These  implicit  models  are  usually  weaker  in  that  they  impose  fewer  constraints. 
They  are  therefore  more  general  and  have  less  predictive  power.  The  distinction  between  weak  and 
strong  models  is  useful.  A  strong  model  imposes  strong  constraints  on  the  system  behavior,  and 
therefore  allows  for  very  efficienct  evaluation.  Its  disadvantage  is  that  it  is  likely  to  be  incorrect.  A 
strong  model,  for  example,  would  have  a  second  order  polynomial  relationship  between  two 
variables.  A  weak  model  would  more  likely  be  correct  but  would  not  allow  as  efficient  evaluation. 
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For  example,  a  very  weak  model  would  only  require  a  monotonic  relationship  between  two 
variables.  This  model  is  less  constraining,  but  makes  less  accurate  predictions. 

Task  Analysis 

Specification  of  a  model  has  further  advantage  in  that  it  requires  a  careful  task  analysis.  The 
analysis  alone  can  frequently  reveal  design  problems  before  a  prototype  is  actually  built.  For  exam¬ 
ple,  task  analysis  can  reveal  whether  the  systems  objectives  are  well  defined.  Objectives  are  neces¬ 
sary  in  order  to  test  and  evaluate  any  system.  Too  often  we  see  evaluation  of  a  system  with  no  par¬ 
ticular  objective  specified  to  the  operatory.  The  generalizability  of  such  results  is  questionable 
because  in  real-world  situations  the  operator  always  has  a  goal  to  accompli.sh. 

In  some  cases,  the  system  is  designed  for  many  purposes  (e.g.,  a  teleoperator  for  extraterrestrial 
applications)  and  it  is  difficult  to  specify  a  single  task.  In  those  situations  one  should  estimate  the 
envelope  of  the  performance  capabilities  and  analyze  several  extreme  tasks.  For  example,  in  the  case 
of  a  teleoperator  design,  one  might  expect  tasks  of  grasping  and  moving  large  objects,  executing  a 
complex  trajectory,  and,  at  the  other  extreme,  inserting  a  small  fastener  into  a  hole. 

One  of  the  results  of  careful  task  analysis  is  the  determination  of  the  variables  to  be  measured. 
Such  analysis  might  reveal  the  need  to  assess  other  measures  than  the  usual  duration  of  a  given  task. 
In  particular,  an  important  by-product  of  a  task  analysis  is  the  notion  that  the  operator  usually  has  a 
choice  of  strategies.  A  typical,  simple  example  is  the  speed-accuracy  trade-off  arising  in  evaluation 
of  various  input  devices.  In  interactions  between  a  human  and  a  machine  most  tasks  can  be  per¬ 
formed  faster  by  sacrificing  accuracy.  A  simple,  weak  model,  of  the  speed-accuracy  trade-off  leads 
to  the  following  evaluation  principles: 

1 .  Always  measure  both  speed  and  accuracy. 

2.  Assure  sufficient  time  pressure  on  the  operator  so  that  their  performance  has  measurable 
errors. 

The  second  principle  assures  that  the  experimenter  can  detect  if  the  operator  changes  strategy 
between  two  conditions  or  two  systems. 

While  I  have  indicated  the  need  for  model-based  evaluation,  it  is  important  to  note  that  using 
models  can  be  very  difficult.  One  argument  frequently  used  against  the  model-based  approach  is  that 
the  validation  of  the  model  is  by  itself  a  complex  empirical  problem  requiring  too  large  an  experi¬ 
mental  effort  to  be  justified.  Although  this  is  a  serious  problem  the  cost  of  evaluation  must  be  com¬ 
pared  to  the  savings  from  the  ability  to  generalize  the  conclusions  and  to  debug  the  system. 

Ruzena  Bajcsy  echoes  the  theme  of  model  based  evaluation  and  task  analysis. 
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Notes  (Ruzena  Bajcsy) 

I  am  most  interested  in  sensory  system  performance.  The  way  I  view  performance  evaluation  is 
that  it  implies: 

Models 

By  models,  I  mean  characterization  of  the  physics  and  geometry  of  the  sensory  system.  Engi¬ 
neers  have  been  doing  modeling  for  a  long  time.  For  example,  for  a  visual  sensor  we  must  model  its 
optics,  absorption  functionl,  resolution,  signal  to  noise  ratio,  etc.  Similarly  for  pressure/force  sensors 
we  should  model  sensitivity,  saturation,  hysteresis,  and  the  like.  For  manipulators,  the  models  give 
degrees  of  freedom,  the  reach-space,  calibration  parameters,  repeatability,  accuracy,  etc.  In  other 
words,  the  models  pro  vice  for  each  parameter  the  limits  of  performance.  What  is  not  so  obvious  is 
that  similar  modeling  principles  must  apply  to  software  modules  as  well  as  to  hardware  modules. 
Examples  of  these  software  modules  are:  convolution  (parameter  is  width  and  orientation  of  the  fil¬ 
ter),  edge-detector  (looking  for  the  first  derivative  or  second  derivative,  the  parameter  is  threshold  of 
acceptance  of  the  peak  in  tht  first  derivative  or  second  derivative),  various  fitting  procedures  to  fit 
surfaces,  volumes  and  so  forh.  This  is  also  related  to  the  choice  of  primitives  which  serve  as  data 
reduction  mechanisms,  and  which  are  necessary  for  displays,  and/or  other  communication  means. 
These  primitives  are  in  fact  more  sophisticated  “sensors”  implemented  in  software. 

Models  of  the  Task 

While  models  of  the  sensors  tell  us  what  CAN  be  measured  and  detected,  a  model  of  the  task  will 
tell  lus  what  NEEDS  to  be  measured.  The  bottom  line  is,  that  by  the  choice  of  models,  one  artifi¬ 
cially  constrains  the  problem,  but  this  is  the  only  way  to  have  a  predictable  system  where  you  take  as 
much  data  as  you  NEED,  determined  by  the  TASK,  and  you  explain  the  data/measurements  as  much 
as  the  models  allow.  So  evaluation  of  a  system  is  always  in  terms  of  some  a  priori  chosen  models. 

Antal  Bejezy  described  specific  issues  which  need  to  be  evaluated  for  teleoperator  systems  in 
space  applications.  He  also  emphasized  task  analysis  and  decomposition  and  an  understanding  of  the 
target  user. 

Measurement  and  Evaluation  in  Teleoperator  Systems  (Antal  Bejezy) 

Virtual  environments  for  teleoperator  control  will  play  an  increasingly  important  role  as  the  sys¬ 
tem  and  task  complexity  and  telecommunication  time  delay  increase.  Preview  and  predictive  graph¬ 
ics  displays  for  task  planning  and  time-delayed  control  execution  can  create  an  essentially  virtual 
environment  for  the  operator.  When  the  graphics  images  of  the  manipulator  and  eventually  also  of 
elements  of  the  task  environment  are  overlayed  over  the  real  TV  images  of  the  manipulator  and  task 
environment,  the  performance  capabilities  of  this  human-machine  interface  technique  will  essentially 
depend  on  the  fidelity  cf  calibration  by  which  the  graphics  images  axe  iclaled  lii  ilit-  teal  images.  This 
is  an  objective  measure,  and  depends  on  the  knowledge  of  a  set  of  system  parameters  (TV  camera 
paramete-'s),  on  mathematical  calibration  techniques,  and  on  the  resolution  of  graphics  images.  The 
higher  the  calibration  fidelity,  the  higher  is  the  operator’s  confidence  in  the  preview  and  predictive 
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power  of  the  graphics.  Through  high-fidelity  calibration  techniques,  the  graphics  images  of  manipu¬ 
lator  and  task  environment  can  truly  become  “virtual  realities”  to  the  operator. 

Graphics  displays  of  abstractions  in  teleoperator  control  are  essentially  visualizations  of  invisible 
events  or  situations.  Such  events  or  situations  are:  forces  exerted  by  manipulator  on  objects  and  on 
environment;  workspace  constraints  or  “holes”  in  a  workspace  not  accessible  under  certain  task 
conditions;  etc.  They  provide  guidance  to  the  operator.  Their  performance  essentially  depends  on 
their  formats  and  on  the  easy  mapping  of  the  display  format  to  the  task  environment.  Here,  pointed 
simplicity  plays  a  key  role.  The  development  of  such  displays  is  in  an  embryonic  state. 

In  ntany  cases,  human- machirie  Imciutcc  devices  and  techniques  in  teleoperation  are  evaluated  as 
separate  entities  or  “subsystems,”  and  their  full-valued  performance  is  predicted  from  such  isolated 
evaluations.  Such  evaluation  methods  are  useful  when  different  interface  devices  and  techniques  are 
compared.  However,  a  specific  device  or  technique  best  reveals  its  virtues  or  weaknesses  when  eval¬ 
uated  in  a  full  system  operation  setting. 

In  many  cases,  the  “naive  user”  is  used  as  a  “reference  operator”  forjudging  or  evaluating  a 
human-machine  interface  device  or  technique  in  teleoperator  control.  Though  this  idea  has  a  well- 
motivated  intention,  care  should  be  taken  to  develop  a  more  tangible  definition  of  the  “naive  user”  as 
a  “reference  operator.”  Probably  what  we  mean  by  a  “naive  user”  when  we  judge  and  evaluate  a 
human-machine  interface  device  or  technique  is:  the  “natural  appeal”  of  the  technique,  the  easy-to- 
remember  features  of  the  device,  the  amount  of  learning  and  training  needed  to  be  fully  familiar  with 
the  device  and  technique,  and  so  on.  The  more  advanced  human-machine  interfaces  do  indeed 
require  familiarization  and  training  before  their  actual  use.  In  particular,  when  teleoperation  becomes 
more  and  more  computer  aided,  the  control  station  wiU  acquire  a  more  and  more  “abstract”  ambience 
which  will  have  to  be  mapped  to  the  mechanical  world. 

The  case  of  multiple  operators  operating  a  teleoperator  system  received  little  attention  from  the 
viewpoint  of  performance  measurements  and  evaluation  methods,  though  it  is  a  fact  that  many  tele¬ 
operator  systems  are  operated  by  at  least  two  operators.  This  is  particularly  true  for  dual-arm  teleop¬ 
erator  systems.  The  obvious  questions  are:  workload  and  function  distribution,  (pilot-copilot;  driver- 
navigator;  etc.),  and  the  distribution  of  responsibilities.  A  good  evaluation  method  for  multiple  oper¬ 
ator  teleoperated  systems  could  also  pave  the  way  for  defining  ways  to  convert  such  systems  into 
single-operator  systems. 

Beth  Marcus  discussed  the  notion  that  evaluations  must  be  task  and  performance  oriented  rather 
than  simply  based  on  an  abstract  notion  oi^  “telepresence”  or  fidelity  to  the  real  world. 

Comment*:  on  Evaluation  (Beth  Marcus) 

The  bottom  line  from  the  panel  discussion  is  that  evaluation  of  human  interfaces  is  not  easy.  The 
suitability  of  a  given  interface  for  teleoperators  or  virtual  environments  is  often  very  task  specific. 
However,  it  is  possible  to  begin  to  collect  a  “wish  list”  for  these  devices.  All  other  things  being  equal 
the  accuracy  of  sensing  should  be  maximized,  training  time  minimized,  and  task  performance  maxi¬ 
mized.  In  addition,  the  trade-off  between  performance  (i.e..  speed,  accuracy)  and  ♦’•aining  time 
should  be  considered. 
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It  is  important  to  note  that,  since  no  interface  hardware  today  or  in  the  foreseeable  future  pro¬ 
vides  all  types  of  feedback  and  gives  a  feeling  identical  to  “being  there,”  we  must  throw  away  the 
notion  that  “natural”  equals  good.  In  a  non-ideal  world,  one  might  be  able  to  convey  more  informa¬ 
tion  in  a  more  useful  form  using  an  artificial  presentation.  For  example,  human  tactile  sense  gives  no 
information  about  proximity,  though  it  may  be  easy  and  useful  to  convey  this  information  through  a 
tactile  display. 

In  order  to  rapidly  expand  the  utility  of  virtual  environments,  we  need  to  take  some  of  the  tech¬ 
nology  that  exists  in  laboratories  today  and  commercialize  it  to  solve  real-world  problems.  Thus,  by 
providing  low  cost  interfaces  which  advance  beyond  the  capabilities  of  a  mouse  or  a  keyboard,  we 
can  have  10,000  or  10  million  people  contributing  to  the  knowledge  base  rather  than  a  few  hundred. 
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PANEL  DISCUSSION  ON  DESIGN  PRINCIPLES  AND  PREDICTIVE 

MODELS 


H.  L.  Alexander,  Chair  (MIT),  N.  Hogan  (MIT),  J.  M.  Hollerback  (McGill  University), 

I.  Hunter  (McGill  University),  and  W.  Verplank  (I.  D.  TWO) 


Questions  Discussed: 

1 .  When  should  master  compliance  be  stiff  and  when  soft?  When  should  slave  compliaxirc  be 
stiff  and  when  soft?  What  are  the  appropriate  places  for  compliance?  When  should  dampling 
be  added  to  master  or  slave? 

2.  When  is  it  necessary  to  have  an  intelligent-computer-mediated  telerobot,  and  when  will  direct 
teleoperation  suffice?  What  functions  are  appropriate  to  the  computer  mediator? 

3.  What  machine  sensory  capabilities  are  important,  in  order  to  support  increased  computer 
participation  in  manipulation?  Conversely,  what  remote  human  sensory  modes  are  important, 
and  how  best  to  present  sensory  data  to  humans? 

4.  Do  current  robot  designs  tend  to  be  too  anthropomorphic,  and  how  should  we  work  to  design 
the  task  (and  environment)  for  the  sake  of  the  robot  as  opposed  to  designing  the  robot? 

The  seminar  opened  with  comments  b>  the  panel  participants.  Bill  Verplank  introduced  the  use 
of  metaphors  to  help  drive  the  design  process.  As  an  example,  he  discussed  the  relatively  familiar 
desktop  metaphor  for  the  computer  interface,  introduced  at  Xerox  PARC,  that  has  suggestd  many 
aspects  of  modem  interfaces,  such  as  the  organization  of  files  (documents)  into  sets  (folders)  that  can 
be  handled,  copied,  moved,  and  displayed  analogously  to  file  folders  in  and  on  a  physical  desk. 

Bill  Verplank  introduced  a  further  metaphor,  motivated  by  the  computer’s  functions  that  may  be 
shared  with  clothing,  as  opposed  to  the  desk.  He  noted  that  the  functions  of  clothing  include  physical 
protection,  providing  a  sense  of  belonging  or  membership  (as  in  the  case  of  uniforms  or  school  col¬ 
ors),  and  generating  a  sense  of  satisfaction  in  ones  appearance.  The  point,  again,  is  to  motivate  think¬ 
ing  into  how  a  computer  might  serve  the  same  functions  as  clothing,  and  perhaps  in  ways  similar  to 
clothing. 

Neville  Hogan  addressed  the  adaptation  of  unfamiliar  remote  vehicle  dynamics,  such  as  the 
gravity-free  dynamics  of  a  teleoperated  space  vehicle  or  the  viscosity-dominated  dynamics  of 
underwater  vehicle,  in  order  to  present  the  operator  with  more  natural  dynamics  to  which  he/she  is 
accustomed.  The  point  is  to  adapt  the  dynamics  of  the  remote  environment,  through  computation  and 
control,  to  the  operator,  rather  than  compelling  the  operator  to  adapt  his/her  self  to  the  remote- 
environment  dynamics. 
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Neville  also  addressed  the  idea  of  turning  manipulator  dynamics  to  tasks,  pointing  out  that 
humans  actually  do  this  relatively  seldom,  and  generally  by  a  modification  of  posture  (for  example, 
in  increasing  stiffness  by  strengthening  the  arm).  Antal  Bejczy  spoke  up  to  defend  the  use  of  selec¬ 
tive  compliance  in  the  performance  of  manipulation  tasks. 

John  Hollerbach  spoke  on  the  design  of  manipulators,  both  macro-  and  micro-,  stressing  the 
advantages  of  parallel-link  manipulators  in  terms  of  their  motion-error  behavior  and  their  stiffness. 
He  pointed  out  that  in  a  parallel-link  manipulator,  individual  errors  tend  to  average  rather  than  to  add 
as  in  the  case  of  serial-link  manipulators,  leading  to  an  overall  decrease  in  positioning  errors. 

Tom  Sheridan  responded  to  this,  mentioning  the  differing  characteristics  of  the  Jacobian  matrices 
corresponding  to  parallel-  and  serial-link  manipulators.  He  went  on  to  introduce  a  new  topic,  and  a 
new  term,  having  to  do  with  the  design  of  human-machine  interface,  for  virtual  and  teleoperator 
interfaces.  He  coined  the  term  “haptic  teleproprioceptive  psychophysics”  to  refer  to  the  detailed 
study  of  human  perception  of  manipulation  cues  that  result  from  contact  with  objects.  He  pointed  out 
that  it  is  important  to  understand  the  details  of  human  proprioceptive  responses,  so  that  we  can  take 
full  advantage  of  that  understanding  in  delivering  those  cues  to  the  operator,  as  though  force-reflect¬ 
ing  hand  controlle*^  *  He  pointed  out  that  input  from  persons  studying  that  field,  and  their  participa¬ 
tion  in  meetings  such  as  this  one,  might  greatly  benefit  teleoperation  and  virtual  environment  studies. 

Ruzena  Bajcsy  emphasized  the  need  for  much  further  sensor  development,  to  allow  for  the 
detection  in  the  remote  environment,  of  tactile  information  for  reflection  to  the  operator.  This  led  lo  a 
discussion  of  sensor  and  actuator  design,  with  particular  reference  to  micromachining  and  its  poten¬ 
tial  in  the  development  of  tactile  sensors  and  displays.  The  group  discussed  micromachining  and  its 
potential  for  creating  actuators  able  to  exert  the  forces  and  displacements  necessary  to  exrite  human 
touch  sensors.  Handayam  SriniVasan’s  work  in  tactile  perception  sensitivity  (which  shows  that 
human  touch  is  indeed  very  sensitive)  indicates  that  such  actuators  may  not  be  too  far  out  of  reach. 

This  discussion,  focusing  as  it  did  on  the  precision  of  feedback  to  individual  fingers  from  a  dex¬ 
trous  remote  manipulator,  led  to  discussion  of  alternatives,  particularly  of  specialized  remote  tools  to 
be  used  in  place  of  general  purpose  remote  manipulators.  Nat  Durlach  prompted  lively  discussion  by 
inviting  designs  of  a  manipulator  that  improve  on  the  human  hand.  Two  such  were  mentioned  - 
canoe  carvers  with  no  pinkie  fingers  to  get  in  the  way,  and  pianists  who  have  cross  connections  in 
their  digital  tendons  severed  in  order  to  improve  the  independent  mobility  of  their  fingers.  Unfortu¬ 
nately,  neither  of  these  manipulators,  however  improved  over  the  originals,  lend  themselves  to 
robotic  applications  at  the  current  time. 

Such  consideration,  however,  led  to  the  discussion  of  the  use  of  muscle  fibers  as  robotic  drive 
elements.  I'his,  in  turn,  introduced  discussion  of  the  phase-transition  (memory)  characteristics  of 
Nitinol  wire  in  the  development  of  actuators  displaying  high  stiffness  and  force  capabilities.  It  was 
also  suggested  that  skin  grown  in  vitro  might  be  used  for  tactile  sensing  with  afferent  nerves  appro¬ 
priately  connected  to  electronic  sensors  and  the  resulting  ner'/e  impuloes  processed  by  neuial 
networks. 

With  regard  to  specialized  tools  replacing  general-purpose  dextrous  end-effectors,  the  point  wa« 
made  that  individual  tools  may  be  developed  that  are  much  better  adapted  to  specific  manipulation 
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and  other  tasks,  then  are  currently  feasible  dextrous  manipulators.  A  general-purpose  tool  interface, 
rather  than  a  dextrous  end-effector,  would  serve  to  attach  each  such  tool  to  the  manipulator  wrist  and 
to  conduct  the  appropriate  interface  services  and  signals.  One  such  swappable  end-effector  would  of 
course  be  a  semi-dextrous  hand,  ranked  next  to  the  drill,  grinder,  wrench,  saw,  and  nut  driver.  The 
chairman  proposed  developing  a  general-purpose  operator  “tele-handle”  for  a  teleoperator  master 
arm,  that  would  provide  a  general-purpose  interface  to  remote  or  virtual  tool,  and  could  supply  a 
wealth  of  tactile  and  other  feedback  to  the  operator. 

Teleoperator  tactile  feedback  having  been  well-discussed,  the  subject  of  tactility  in  virtual  worlds 
was  addressed.  Whereas  in  teleoperator  interfaces  to  remote,  physical  worlds  the  problem  includes 
both  sensing  (remotely)  and  generating  (locally)  tactile  stimuli,  in  virtual  worlds  the  sensing  problem 
is  eliminated.  The  problem  becomes  instead  one  of  synthesizing  tactile  stimuli,  corresponding  either 
literally  (as  in  the  case  of  contact  with  a  hand,  unyielding  virtual  wall)  or  metaphorically  (what  does 
it  feel  like  to  have  ones  hand  pa<.s  through  a  virtual  wall?)  to  the  virtual  reaUty. 

*Chairman’s  note:  With  our  increased  understanding  of  and  capability  for  tactile  feedback,  we 
may  want  to  generalize  the  term  “force-reflecting”  to  sense-reflecting  hand-controllers.  An  example 
is  Steve  Jacobson’s  controller  which  delivers  the  tap  responses  to  the  operator  on  contact  with 
objects,  which  response  is  not  merely  derived  from  force  feedback,  but  is  deliberately  synthesized  to 
represent  to  the  operator  the  contact  event. 
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MINI  SEMINAR:  TELEOPERATORS  WITH  MICRO  SLAVE  ROBOTS 


K.  J.  Gabriel,  Chair 
University  of  Tokyo 
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POSTER  PRESENTATIONS 


THE  HUMAN  OPERATOR  TRANSFER  FliNCTION:  U)ENTIFICATION  OF 

THE  LIME  MECHANICS  SUBSYSTEM 


Lynette  A.  Jones  and  Ian  W.  Hunter 
McGiil  University 
Montreal,  Quebec,  CANADA 

In  most  studies  of  human  manual  control  systems,  the  operator  is  characterized  as  a  lumped, 
indivisible  system,  whose  dynamics  are  linear  and  of  low  order.  The  objective  of  our  research  is  to 
decompose  the  performance  of  the  human  operator  in  terms  of  the  subsystems  that  determine  tlie 
operator’s  responses  in  order  to  establish  how  the  dynamics  of  these  component  subsystems  influ¬ 
ence  the  operator’s  performance.  In  the  present  experiment,  the  dynamic  stiffness  of  the  human 
elbow  joint  was  measured  at  rest  and  under  different  levels  of  biceps  muscle  activation;  this  work 
forms  part  of  the  analysis  of  the  limb  mechanics  subsystem. 

The  stiffness  of  the  elbow  joint  was  determined  by  subjecting  the  right  wrist  to  stochastic  posi¬ 
tion  perturbations  (S.D.  3mm,  bandwidth  of  lOOHz)  delivered  by  an  electromagnetic  linear  motor 
(100  mm  maximum  linear  displacement,  peak  static  force  500  N).  Subjects  generated  a  target  force 
(ranging  from  0  to  150  N)  during  the  perturbations,  and  were  provided  with  visual  feedback  of  the 
mean  force  produced. 

The  stiffness  frequency  response  function  was  determined  at  each  mean  force  level.  These  func¬ 
tions  were  calculated  by  Fourier  transforming  the  stiffness  impulse  response  functions.  Figure  1 
shows  the  stiffness  frequency  response  functions  obtained  from  a  subject  generating  a  mean  force  of 
25N  (thin  line)  and  150N  (thick  line).  The  increase  in  the  low  frequency  stiffness  with  an  increase  in 
mean  muscle  activation  level  (as  reflected  in  the  mean  force)  is  clearly  evident.  The  figure  also 
demonstrates  that  stiffness  increases  at  higher  frequencies.  This  essentially  high-pass  characteristic 
of  the  limb  mechanics  system  is  well-known.  In  Figure  2,  a  typical  coherence  function  is  shown.  The 
decrease  in  coherence  at  higher  frequencies  is  due,  in  part,  to  a  loss  in  the  signal-to-noise  ratio 
cau.sed  by  limitations  in  the  bandwidth  of  the  motors,  whereas  the  loss  of  coherence  at  lower  fre¬ 
quencies  probably  reflects  the  contribution  of  reflexes. 
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TACTILE  DISCRIMINATION  AND  REPRESENTATIONS  OF  TEXTURE, 

SHAPE,  AND  SOFTNESS 


M.  A.  Srinivasan 
Research  Lab  of  Electronics 
Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts 

and 


R.  H.  LaMotte 

Department  of  Anesthesiology 
Yale  University  School  of  Medicine 
New  Haven,  Connecticut 


We  present  here  some  of  the  salient  results  on  the  tactual  discriminabUities  of  human  subjects 
obtained  through  psychophysical  experiments,  and  the  associated  peripheral  neural  codes  obtained 
through  electrophysiological  recordings  from  monkey  single  nerve  fibers.  Humans  can  detect  the 
presence  of  a  2  microns  high  single  dot  on  a  smooth  glass  plate  stroked  on  the  skin,  based  on  the 
responses  of  Meissner  type  rapidly  adapting  fibers  (RAs).  They  can  also  detect  a  0.06  microns  high 
grating  on  the  plate,  owing  to  the  response  of  Pacinian  corpuscle  fibers.  Among  all  the  possible  rep¬ 
resentations  of  the  shapes  of  objects,  the  surface  curvature  distribution  seems  to  be  the  most  relevant 
for  tactile  sensing.  Slowly  adapting  fibers  respond  to  both  the  change  and  rate  of  change  of  curvature 
of  the  skin  surface  at  the  most  sensitive  spot  in  their  receptive  fields,  whereas  RAs  respond  only  to 
the  rate  of  change  of  curvature.  Human  discriminability  of  compliance  of  objects  depends  on 
whether  the  object  has  a  deformable  or  rigid  surface.  When  the  surface  is  deformable,  the  spatial 
pressure  distribution  within  the  contact  region  is  dependent  on  object  compliance,  and  hence  infor¬ 
mation  from  cutaneous  mechanoreceptors  is  sufficient  for  discrimination  of  subtle  differences  in 
compliance.  When  the  surface  is  rigid,  kinesthetic  information  is  necessary  for  discrimination,  and 
the  discriminability  is  much  poorer  than  that  for  objects  with  deformable  surfaces. 
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A  fflGH  PERFORMANCE  TWO  DEGREE-OF-FREEDOM  KINESTHETIC 

INTERFACE 


Bernard  D.  Adelstein  and  Michael  J.  Rosen 
Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts 


ABSTRACT 


This  sununary  focuses  on  the  kinesthetic  interface  of  a  virtual  invironment  system  that  was 
developed  at  the  Newman  Laboratory  for  Biomechanics  and  Human  Rehabilitation  at  M.I.T.  for  the 
study  of  manual  control  in  both  motorically  impaired  and  able-bodied  individuals. 


MECHANISM  DESIGN 


The  kinesthetic  interface  is  an  electromechanically  actuated  manipulandum  (i.e.,  backdriveable 
manipulator)  that  enables  dynamic  interaction  between  the  arm  of  a  human  operator/experimental 
subject  and  controlled  mechanica’  loads.  The  manipulanduncu  depicted  in  Figure  1,  consists  of  a 
direct-drive  handle  linkage  which  is  coupled  to  two  printed  armature  brushed  DC  motors.  The  inter¬ 
face  is  instrumented  to  sense  displacements,  velocities,  and  acceleratious  at  the  motor  shafts,  and 
forces  between  the  hand  grip  and  the  linkage. 

The  coupling  mechanism  that  joins  the  handle  to  the  motors  forms  a  spherical  closed-chain  five- 
bar  linkage,  resulting  in  two  kinematic  degrees  of  freedom  at  the  hand  grip.  This  linkage  permits 
both  actuator  housings  to  be  anchored  to  a  common  mechanicsl  ground,  thereby  making  it  unneces¬ 
sary  to  carry  their  excessive  weight  and  inertia.  Because  the  linkage  consists  of  only  ball  bearing 
revolute  pairs  and  rigid  links,  undesirable  friction,  backlash,  and  compliance  characteristics  associ¬ 
ated  with  other  transmission  types  are  greatly  reouced  in  this  interface. 

Additional  attributes  of  this  mechanism  configuration  prove  advantageous  for  manipulandum 
control.  Over  its  17  cm  square  workspace,  the  interface  is  approximately  planar,  and,  most  impor¬ 
tantly,  the  actions  of  the  two  motors  are  nearly  decoupled. 


SYSTEM  CONTROL 


The  overall  virtual  environment  system,  shown  in  Figure  2,  consists  of  the  kinesthetic  interface 
and  a  planar  video  display.  System  operation  is  supervised  by  an  LSI-1 1/23  based  micro  computer, 
responsible  for  collecting  and  storing  data,  commanding  and  monitoring  the  Control  Interface  Unit 
(CIU),  and  transferring  target  and  tracking  response  information  to  the  Amiga  1000  personal 
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computer  for  display  to  the  subject.  The  CIU  contains  hardwired  circuitry  to  control  load  simulations 
by  each  motor  as  well  as  maintain  system  safety. 

Because  of  the  manipulandum’s  nearly  planar  and  nearly  decoupled  linkage  kinematics,  configu¬ 
ration  space  control  can  be  used  to  regulate  endpoint  impedance  directly,  without  geometric  compu¬ 
tation.  As  a  consequence,  the  LSI- 1 1/23  host  can  update  CIU  load  control  parameters  at  rates 
exceeding  1  kHz. 


MECHANICAL  LOAD  AND  OBJECT  SIMULATION 


The  manipulandum’s  ability  to  produce  positive  (i.e.,  resistive)  and  negative  {i.e.,  destabilizing) 
spring  and  damping  fields  was  examined.  Load  ranges  achieved  with  the  CIU’s  hardwired  digitally- 
supervised  analog  controller  are  listed  in  Table  1.  Greater  stiffness — up  to  3  times  higher — are 
obtainable  under  purely  digital  control  implementations. 


Load  Typo 

Rotational 

Translational  Equivalent 

Active  Resistive  Stiffness 
Active  Resistive  Damping 
Active  Inertia 

Maximum  Torque/Force 

0-136  N-m/rad 

0-6.7  N-m/rad/s 
.007-0.045  kg-m2 

4.7  N-m 

0-3100  N/m 

0-152  N/m/s 
.13-1  kg 

21  N 

Table  1;  Load  ranges  achieved  under  digitally  supervised  analog  control.  The  translational  load 
equivalent  is  calculated  for  the  hand  grip  end  of  Ae  fixed  length  handle. 

In  addition  to  simulating  mechanical  impedances,  the  controller  can  either  mask  or  enhance  the 
manipulandum’s  passive  inertia  and  friction,  thereby  allowing  the  handle  to  feel  “invisible”  when 
pushed  and  have  sufficient  friction  to  stay  in  place  when  “let  go”  by  the  human  operator.  Contact 
with  a  hard  non-sticky  wall  anc  capture  by  a  detent  also  have  been  simulated  convincingly  by  the 
manipulandum,  as  illustrated  in  Figures  3  and  4.  Together,  the  performance  capabilities  described  in 
this  paragraph  successfully  meet  a  set  of  criteria  used  in  the  industry  to  demonstrate  that  an  “artificial 
control  feel”  device  is  versatile  enough  to  faithfully  execute  any  general  simulation  (Jex  1988). 
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Figure  1 .  The  manipulandum.  Height  from  floor  to  spherical  hand  grip  is  1  meter. 
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Figure  2.  Virtual  environment  system  components  and  communication.  The  highlighted  frequencies 
{clockwise  from  upper  left )  indicate  the  computer  update  rates  for  the  visual  display  and  load 
controller,  and  the  manipulandum  force  bandwidth  as  determined  by  its  first  structural  mode. 


Ill 
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Figure  3.  Wall  model  and  implementations.  The  wall  is  modelled  {left )  as  a  massless  plate  backed 
by  a  stiff  spring  and  viscous  damper.  When  the  handle  is  to  the  right  of  the  wall  rest  position  {top  ), 
no  resistance  is  encountered.  As  the  handle  moves  past  the  wall  rest  position  {bottom  ),  the  spring 
and  dampers  oppose  further  motion  to  the  left.  Torques  resulting  from  the  hand  force  applied  to  the 
constant  length  manipulandum  handle  are  plotted  against  displacement  {right )  to  demonstrate  the 
effect  of  bandwidth  on  wall  quality.  The  upper  plot  shows  wdl  performance  with  a  20  msec  lag 
introduced  prior  to  detection  of  the  handle  position  by  the  computer.  The  lower  plot  has  this  lag 
removed.  Both  simulations  are  updated  at  IkHz.  The  lag  causes  a  perceived  “stickiness,”  visible  in 
the  upper  plot  as  the  attractive  force  pulling  the  handle  back  toward  the  wall  {i.e.,  displacement  =  0) 
Away  from  the  wall,  the  difference  between  mean  force  levels  during  approach  and  withdrawal 
motions  is  due  to  uncompensated  friction. 
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Figure  4.  Detent  model  and  implementation.  The  detent  is  modelled  {top )  as  a  spring  loaded  ball 
plunger  located  at  the  bottom  of  the  manipulandum  handle  that  is  trapped  when  it  passes  over  the 
indentation  in  the  planar  workspace.  The  plot  shows  the  history  of  torque  due  to  handle  force  versus 
handle  displacement  in  one  of  the  manipulandum  degrees  of  freedom  as  the  handle  is  captured  by  a 
simulated  detent,  comes  to  a  stop,  and  then  breaks  free. 
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PERCEPTUAL  AND  PERFORMANCE  CONSEQUENCES  OF 
FLIGHT  IN  VIRTUAL  WORLDS 


C.  Thomas  Bennett  and  Martin  Schwirzke 
NASA  Ames  Research  Center 
Moffett  Field,  California 

and 


James  S.  Tittle 
University  of  California 
Irvine,  California 


Head-mounted,  virtual  world  technologies  were  developed  in  the  1960s.  Comeau  and  Bryan 
(1961)  reported  on  a  head-slaved  video  control  system.  In  1968,  Sutherland  discussed  a  head-slaved, 
graphics  based  virtual  world  that  was  later  demonstrated  by  Vickers  (1970).  Operational  systems  for 
military  aircraft  were  quickly  developed  and  tested  (Johnson  and  Foster,  1977). 

As  part  of  second  generation  technology  efforts,  new  display  media  and  computer  graphics  sys¬ 
tems  have  been  developed  (Fisher,  McGreevey,  Humphries,  and  Robinett,  1986).  With  this  new 
technology  effort  has  come  more  systematic  studies  of  the  performance  consequences  resulting  from 
conducting  teleoperations  in  aeronautical  settings. 

There  are  two  primary  purposes  for  head-mounted  systems  in  aeronautical  settings.  One  is  for 
helmet-mounted  sights  and  teleoperated  (head-slaved)  weapons  systems.  Bennett,  Johnson,  Perrone, 
and  Phatak  (1988)  evaluated  head  tracking  performance  during  passive  and  controlled  flight.  In  that 
study,  comparisons  were  also  made  of  head  tracking  performance  in  sterile  and  relatively  complex 
virtual  worlds.  That  study  confirmed  the  robusmess  of  head  tracking  performance  across  a  wide 
variety  of  visual  scenes. 

A  second  use  of  virtual  world  displays  is  for  aircraft  control.  Aircraft  controllability  using  head- 
mounted,  panel-mounted  or  simulated  out-the-window  scenes  has  been  systematically  examined 
(Bennett,  O’Donnell,  and  Johnson,  1988;  O’Donnell,  Johnson,  and  Bennett,  1988).  Those  studies 
reported  the  range  of  rotorcraft  flight  tasks  in  which  head-mounted  virtual  worlds  provided  some 
advantages. 

Two  studies  will  be  reported  that  examine  the  perceptual/performance  effects  of  virtual  worlds 
The  first  examines  head  tracking  performance  with  roll-stabilized  versus  non-roll  stabilized  virtual 
worlds.  The  purpose  of  the  study  was  to  (a)  examine  display  strategies  used  in  current  display  sys¬ 
tems  and  (b)  study  the  adaptability  cf  obsen/ers  to  estimated  glide  slope  angles  usi^'g  head  sla'.  cd 
versus  head-stabilized  imagery.  The  purpose  of  this  study  was  to  examine  the  usefulness  of  wide 
field-of-regards  during  final  approaches  to  a  runway. 
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A  DECADE  OF  TELEROBOTICS  IN  REHABILITATION:  DEMONSTRATED 
UTILITY  BLOCKED  BY  THE  HIGH  COST  OF  MANIPULATION  AND  THE 
COMPLEXITY  OF  THE  USER  INTERFACE 


Larry  Leifer,  Stefan  Michalowski,  and  Machiel  Van  Der  Loos 
Stanford  University 
Stanford,  California 


The  StanfordA^A  Interactive  Robotics  Laboratory  set  out  in  1978  to  tect  the  hypothesi:;  that 
industrial  robotics  technology  could  be  applied  to  serve  the  manipulation  needs  of  severely  impaired 
individuals.  Five  generations  of  hardware,  three  generations  of  system  software,  and  over  125  exper¬ 
imental  subjects  later,  we  believe  that  genuine  utility  is  achievable.  The  experience  includes  devel¬ 
opment  of  over  65  task  applications  using  voiced  conunand,  joystick  control,  natural  language  com¬ 
mand  and  3D  object  designation  technology. 

A  brief  foray  into  virtual  environments,  using  flight  simulator  technology,  was  instructive.  If 
reality  and  virtuality  come  for  comparable  prices,  you  cannot  beat  reality. 

A  detailed  review  of  assistive  robot  anatomy  and  the  performance  specifications  needed  to 
achieve  cost/beneficial  utility  will  be  used  to  support  discussion  of  the  future  of  rehabilitation  teler¬ 
obotics.  Poised  on  the  threshold  of  commercial  viability,  but  constrained  by  the  high  cost  of  techni¬ 
cally  adequate  manipulators,  this  worthy  application  domain  flounders  temporarily.  In  the  long  run, 
it  will  be  the  user  interface  that  governs  utility. 
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A  PHYSICAL  CONTROL  INTERFACE  WITH  PROPRIOCEPTIVE 
FEEDBACK  AND  MULTIPLE  DEGREES  OF  FREEDOM 


G.  H.  Creasey,  D.  Gow,  Y.  Sloan,  and  B.  Meadows 
University  of  Edinburgh  and  Princess  Margaret  Rose  Orthopaedic  Hospital 

Edinburgh,  SCOTLAND 


The  use  of  the  drug  thalidomide  by  pregnant  mothers  in  Britain  resulted  in  a  variety  of  deformi¬ 
ties  including  the  birth  of  children  having  no  arms.  Such  children  were  provided  with  powered  artifi¬ 
cial  arms  with  up  to  five  degrees  of  freedom  simultaneously  controlled  in  real  time  by  shoulder 
movement  (ref.  1)  and  whose  operation  could  be  learnt  by  children  within  a  matter  of  hours.  The 
ease  with  which  this  manipulation  could  be  learnt  and  used  may  have  been  due  to  the  system  used  to 
provide  feedback  of  position  and  force  to  the  user’s  skin  and  joints.  In  this  way,  the  physiological 
sense  of  proprioception  was  extended  from  the  user  into  the  device,  reducing  the  need  for  visual 
feedback  and  conscious  control. 

With  the  banning  of  thalidomide,  this  technique  fell  into  disuse  but  it  is  now  being  re-examined 
as  a  control  mechanism  for  other  artificial  limbs  (refs.  1-5)  and  it  may  have  other  medical 
applications  to  allow  patients  to  control  formerly  paralysed  limbs  moved  by  electrical  stimulation 
(ref.  6).  It  may  also  have  commercial  applications  in  robotic  manipulation  or  physical  interaction 
with  virtual  environments. 

To  allow  it  to  be  investigated  further,  the  original  pneumatic  control  system  has  recently  been 
converted  to  an  electrical  analogue  to  allow  interfacing  to  electronic  and  computer-assisted  systems. 
A  harness  incorporates  force-sensitive  resistors  and  linear  potentiomenters  for  sensing  position  and 
force  at  the  interface  with  the  skin,  and  miniature  electric  motors  and  lead  screws  for  feeding  back  to 
the  user  the  position  of  the  robotic  arm  and  the  forces  applied  to  it.  In  the  present  system,  control  is 
applied  to  four  degrees  of  freedom  using  elevation/depression  and  protraction/retraction  of  each 
shoulder  so  that  each  collar-bone  emulates  a  joystick.  However,  both  electrical  and  mechanical 
components  have  been  built  in  modular  form  to  allow  rapid  replication  and  testing  of  a  variety  of 
force  and  position  control  strategies. 
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"ROTATIONAL  WIND”  INDICATOR  ENHANCES  CONTROL  OF 
ROTATED  DISPLAYS:  H.  A.  CUNNINGHAM  AND  M.  PAVEL 


H.  A.  Cunningham  and  M.  Pavel 
Stanford  University 
Stanford,  California 

and 

NASA  Ames  Research  Center 
Moffett  Field,  California 


Rotation  by  108°  of  the  spatial  mapping  between  a  visual  display  and  a  manual  input  device  pro¬ 
duces  large  spatial  errors  in  a  discrete  aiming  task.  These  errors  are  not  easily  corrected  by  voluntary 
mental  effort,  but  the  central  nervous  system  does  adapt  gradually  to  the  new  mapping.  Bemotat 
(1970)  showed  that  adding  true  hand  position  to  a  90°  rotated  display  improved  performance  of  a 
compensatory  tracking  task,  but  tracking  error  rose  again  upon  removal  of  the  explicit  cue.  This  sug¬ 
gests  that  the  explicit  error  signal  did  not  induce  changes  in  the  neural  mapping,  but  rather  allowed 
the  operator  to  reduce  tracking  error  using  a  higher  mental  strategy. 

In  this  report,  we  describe  an  explicit  visual  display  enhancement  applied  to  a  108°  -rotated  dis¬ 
crete  aiming  task.  A  “wind  indicated*  corresponding  to  the  effect  of  the  mapping  rotation  is  dis¬ 
played  on  the  operator-controlled  cursor.  The  human  operator  is  instructed  to  oppose  the  virtual 
force  represented  by  the  indicator,  as  one  would  do  if  flying  an  airplane  in  a  crosswind.  This 
enhancement  reduces  spatial  aiming  error  in  the  first  10  minutes  of  practice  by  an  average  of  70% 
when  compared  to  a  no  enhancement  control  condition.  Moreover,  it  produces  adaptation  aftereffect, 
which  is  evidence  of  learning  by  neural  adaptation  rather  than  by  mental  strategy.  Finally,  aiming 
error  does  not  rise  upon  removal  of  the  explicit  cue. 
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OBJECTIVE  EVALUATION  OF  SITUATION  AWARENESS  FOR  DYNAMIC 
DECISION  MAKERS  IN  TELEOPERATIONS 


Mica  R.  Endsley 
Northrop  Corporation 
Lomita,  California 


Situation  awareness,  a  current  mental  mode  of  the  environment,  is  critical  to  the  ability  of  opera¬ 
tors  to  perform  complex  and  dynamic  tasks.  This  should  be  particularly  true  for  teleoperators,  who 
are  separated  from  the  situation”  they  need  to  be  aware  of.  The  design  of  the  man-machine  interface 
must  be  guided  by  the  goal  of  maintaining  and  enhancing  situation  awareness. 

The  objective  of  this  work  has  been  to  build  a  foundation  upon  which  research  in  the  area  can 
proceed.  A  model  of  dynamic  human  decision  making  which  is  inclusive  of  situation  awareness  will 
be  presented,  along  with  a  definition  of  situation  awareness.  A  method  for  measuring  situation 
awareness  will  also  be  presented  as  a  tool  for  evaluating  design  concepts.  The  Situation  Awareness 
Global  Assessment  Technique  (SAGAT)  is  an  objective  measure  of  situation  awareness  originally 
developed  for  the  fighter  cockpit  environment.  The  results  of  SAGAT  validation  efforts  will  be  pre¬ 
sented.  Implications  of  this  research  for  teleoperators  and  other  operators  of  dynamic  systems  will  be 
discussed. 
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DEVELOPMENT  OF  VISUAL  3D  VIRTUAL  ENVIRONMENT 
FOR  CONTROL  SOFTWARE 


Michitiika  Hirose,  Takeshi  Myoi,  Haruo  Amari,  Kohei  Inamura,  and  Lawrence  Stark 
University  of  Tokyo  and  Tokyo  Electric  Power  Co.,  Tokyo,  JAPAN 
and  University  of  California,  Berkeley  California 


Virtual  environments  for  software  visualization  may  enable  complex  programs  to  be  created  and 
maintained.  A  typical  application  might  be  for  control  of  regional  electric  power  systems.  As  these 
encompass  broader  computer  networks  than  ever,  construction  of  such  systems  becomes  very  diffi¬ 
cult.  Conventional  iext-orientcd  environments  are  useful  in  programming  individual  processors. 
However,  they  are  obviously  insufficient  to  program  a  large  and  complicated  system,  that  includes 
large  numbers  of  computers  connected  to  each  other;  such  programming  is  called  “programming  in 
the  Uirge.” 


As  a  solution  for  this  problem,  the  authors  are  developing  a  graphic  programming  environment 
wherein  one  can  visualize  complicated  software  in  virtual  3D  world.  One  of  the  major  features  of  the 
environment  is  the  3D  representation  of  concurrent  process.  3D  representation  is  used  to  supply  both 
network-wide  interprocess  programming  capability  (capability  for  “programming  in  the  large”)  and 
real-time  programming  capability.  The  authors’  idea  is  to  fuse  both  the  block  diagram  (which  is  use¬ 
ful  to  check  relationship  among  large  number  of  processes  or  processors)  and  the  time  chart  (which 
is  useful  to  check  precise  timing  for  synchronization)  into  a  single  3D  space.  The  3D  representation 
gives  us  a  capability  for  direct  and  intuitive  planning  or  understanding  of  complicated  relationship 
among  many  concurrent  processes. 

To  realize  the  3D  representation,  a  technology  to  enable  easy  handling  of  virtual  3D  object  is  a 
definite  necessity.  Using  a  stereo  display  system  and  a  gesture  input  device  (VPL  DataGlove),  our 
prototype  of  the  virtual  workstation  has  been  implemented.  The  workstation  can  supply  the 
“sensation”  of  the  virtual  3D  space  to  a  programmer.  Software  for  the  3D  programming  environment 
is  implemented  on  the  workstation.  According  to  preliminary  assessments,  a  50%  reduction  of  pro¬ 
gramming  effort  is  achieved  by  using  the  virtual  3D  environment.  The  authors  expect  that  the  3D 
environment  has  considerable  potential  in  the  field  of  software  engineering. 

PROGRAMMING  IN  THE  LARGE 

The  authors  arc  involved  in  the  design  project  for  future  power  supply  system  in  Tokyo.  The  goal 
of  the  project  is  to  have  the  entire  power  distributing  system  controlled  automatically  by  a  vast  com¬ 
puter  networking  system.  In  order  to  realize  that  goal,  we  need  to  find  ways  to  solve  the  problems 
which  arise  in  developing  such  a  large  and  complicated  software  scheme. 

fwo  basic  types  of  problems  associated  with  the  different  levels  of  the  networking  environment 
can  be  considered  One  type  of  question  might  be:  “How  do  I  program  each  computer?”  (This  is 
called  programming  in  the  small).  Another  mights  be;  “How  do  I  coordinate  many  computers?.” 
(This  IS  called  programming  in  the  large.) 
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Since  the  programming  environment  itself  is  built  on  “algonthmic”  or  “logical”  characteristics,  it 
is  convenient  to  use  text  representation.  With  text,  we  can  eliminate  much  ambiguity.  However,  text 
alone  becomes  limitingly  insufficient  in  the  case  of  “programming  in  the  large”  where  the  most 
important  factor  in  the  field  concerns  total  configuration  de  ign  rather  than  detailed  logic  design. 
Thus  a  more  global  methodology  is  needed  to  correct  for  faults  in  the  overall  system  design,  even 
though  each  indiv  idual  subsystem  may  be  correctly  designed. 

In  “programming  in  the  large,”  the  authors  believe  that  visual  representations  play  an  important 
role  in  the  intuitive  understanding  of  software  and  a  combination  of  text  and  visual  environments 
will  provide  the  best  solution. 

CONCEPT  OF  3D  VISUAL  REPRESENTATION 

Often,  a  network  diagram  is  given  as  a  2D  representation,  known  as  block  diagram.  However,  if 
the  description  of  a  real-time  control  program  is  required,  such  as  in  the  synchronization  of  several 
control  processes,  the  time  dimension  should  also  be  taken  into  consideration. 

Fig.  1  shows  the  concept  of  the  3D  visual  representation.  If  we  observe  the  representation  along 
the  X  axis,  it  will  be  a  conventional  block  diagram.  The  axis  normal  to  the  X  axis  is  a  time  chart. 
Using  this  3D  representation,  the  programmer  can  intuitively  grasp  the  state  of  concurrent  process. 

In  the  3D  visual  representation,  the  geometrical  shape  used  to  represent  program  flow  of  succes¬ 
sive  message  passing  is  also  important.  The  shape  itself  indicates  the  type  of  information  exchange. 
Sometimes  the  programmer  can  handle  the  software  entirely  based  on  its  shape.  The  greater  the 
number  of  processes,  the  greater  the  advantages  of  3D  representation,  presumably  due  to  characteris¬ 
tics  of  the  human  cognitive  processes. 

IMPLEMENTATION 

Several  years  ago,  such  an  idea  was  unrealizable  due  to  the  limitations  of  computer  capabilities. 
To  implement  3D  representations,  we  need  a  new  technology  able  to  incorporates  real-time  3D  ani¬ 
mation,  stereo  displays,  3D  input  devices,  and  other  aspects  of  the  virtual  3D  work-space.  To  gener¬ 
ate  a  3D  image,  the  authors  used  Stereographies  CrystalEyes  System,  (a  flickerless  3D  CRT  by  high 
speed  LC  (120  Hz  Liquid  Crystal)  shutter).  By  adding  a  head  movement  detector  (Polhemus  3D 
tracker)  to  that  system,  a  partial  “look-around”  effect  can  be  implemented.  As  a  gestural  input 
device,  VPL  DataGlove  is  used. 

Fig.  2  shows  the  current  hardware  configuration  of  the  virtual  3D  workstation.  The  system 
includes  two  sub-workstations.  One  of  them  (an  HP9000  SRX)  is  used  for  generating  the  virtual  3D 
image  and  handling  I/O  devices.  Another  (  a  Sun  3)  is  used  for  text  handling.  In  other  words,  the  first 
one  is  for  “programming  in  the  large,”  and  the  second  is  for  “programming  in  the  small.” 

Figure  3  shows  an  example  of  the  visual  programming  environment.  The  message  flow  between 
processes  is  displayed  in  virtual  3D  space.  One  small  box  represents  a  computing  object  (sometimes 
it  can  be  a  process,  sometimes  it  can  be  a  processor). 
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Semi-tranparent  box  represents  a  carrier  of  computing  objects  (it  can  be  a  stand-alone  computer 
box). 

Using  a  virtual  hand  which  works  as  3D  pointer  and  a  kind  of  “action  menu”  which  detects  ges¬ 
tures,  programmer  can  handle  virtual  objects  (such  as  “objects”  and  “messages”)  in  the  virtual  3D 
world.  Currently,  the  environment  has  two  modes,  called  “world  mode”  and  “succession  mode.” 
World  mode  is  mainly  for  observing  whole  shapes  of  the  software.  Succession  mode  is  used  for  edit¬ 
ing  messages  passing  among  objects.  The  configuration  is  still  preliminary.  Further  investigation  is 
needed  concerning  the  design  of  object  handling  in  3D  virtual  space. 

Tools  included  in  the  environment  are  as  follows: 

1.  Virtual  Measure/Ruler;  Tool  to  measure  the  exact  relationship  among  virtual  objects. 

2.  Critical  Path  Finder:  Tool  to  fmd  and  display  the  critical  path  which  determines  the  total 
network  throughput  for  a  given  network  task. 

3.  Network  Simulator:  Tool  to  simulate  message  passing  and  data  processing  on  each  computer. 

4.  Network  Planner:  Tool  to  update  information  about  the  network  configuration.  This  tool 
works  with  Network  Simulator. 

ASSESSMENTS  AND  CONCLUSION 

As  a  preliminary  assessment  of  the  3D  environment,  several  multiprocessor  programming  tasks 
were  assigned  to  subjects.  The  target  system  is  supposed  to  be  3-4  networked  personal  computers. 
Completion  time  with  and  without  the  environment  was  measured.  Roughly  speaking,  completion 
time  with  the  3D  environment  was  shorter  than  that  of  the  conventional  methodology.  In  addition, 
using  3D  lowers  the  variance,  indicating  that  the  environment  reduces  error. 

The  authors  believe  that  the  concept  of  3D  visual  representations  has  proven  to  be  effective  even 
in  the  simple  environment  already  implemented.  A  far  greater  impact  can  be  expected  in  a  more 
sophisticated  environment  as  the  most  case  of  software  environment.  For  example,  SmalltalkSO  is 
highly  valued  for  its  concept  of  “object  oriented”  programming,  but  its  environment  is  valued  much 
more  than  the  concept  itself.  The  authors  consider  this  kind  of  practical  effort  is  indispensable.  From 
a  practical  point  of  view,  however,  much  effort  is  needed  to  prepare  and  accumulate  effective  tools 
for  this  new  environment. 
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Figure  3.  Example  of  visual  3D  programming  environment. 
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HUMAN  OPERATOR  TRACKING  PERFORMANCE  WITH  A 
VIBROTACTILE  DISPLAY 


Gideon  F.  Inbar 

Technion  -  Israel  Institute  of  Technology 
Haifa ,  ISRAEL 


Vibrotactile  displays  have  been  designed  and  used  as  a  sensory  aid  for  the  blind.  In  the  present 
work  the  same  6  x  24  “Optacon”  type  vibrotactile  display  (VTD)  was  used  to  characterize  human 
operator  (HO)  tracking  performance  in  pursuit  and  compensatory  tasks.  The  VTD  was  connected  via 
a  microprocessor  to  a  one-dimensional  joy  stick  manipulator.  Various  display  schemes  were  tested 
on  the  VDT,  and  were  also  compared  to  visual  tracking  performance  using  a  specially  constructed 
photo  diode  matrix  display  comparable  to  the  VTD. 

Optimal  performance,  measured  as  minimal  rms  error  between  input  and  output  signals,  was 
achieved  by  dividing  the  VDT  matrix  into  two  3  x  24  fields.  On  one  side,l  the  reference  signal  was 
presented  and  on  the  second  half  the  target  position.  The  axial  position  on  either  half  was  generated 
by  cycling  between  two  adjacent  3  pin  lines  at  40H2,  moving  up  and  down  with  the  target,  or  refer¬ 
ence  signals.  Pseudo  random  bandlimited  signals,  EX!!  to  0.4HZ  were  used  testing  the  performance  of 
four  subjects. 

Results; 

1 .  Very  little  training  was  needed  by  the  subjects  to  learn  to  use  the  system  and  there  was  little 
improvement  with  time  after  the  quick  initial  learning  phase. 

2.  No  significant  differences  were  measured  between  pursuit  and  compensatory  tasks,  in 
contrast  to  the  visual  tracking  experiment. 

3.  The  calculated  coherence  function  for  both  tasks  was  close  to  1 .0  at  the  above  dynamic 
range. 

4.  The  HO  transfer  function,  after  the  exclusion  of  a  fixed  time  delay,  was  a  constant  gain  and 
constant  phase  at  the  above  range. 

5.  An  average  of  about  420  msec,  time  delay  between  the  input  and  output  signals  was 
exhibited  by  three  subjects  while  the  fourth  subject  had  almost  double  this  delay. 
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IMPLEMENTATION  ISSUES  FOR  A  COMPACT  6  DEGREE  OF  FREEDOM 
FORCE  REFLECTING  HANDCONTROLLER 
WITH  CUEING  OF  MODES 


Heidi  Jacobus,  Alan  J.  Riggs,  Charles  Jacobus,  and  Yechiel  Weinstein 
Cybernet  Systems  Corporation 
Ann  Arbor,  Michigan 


Teleoperated  control  requires  a  master  human  interface  device  that  can  provide  haptic  input  and 
output  which  reflects  the  responses  of  a  slave  robotic  system.  The  effort  reported  in  this  paper 
addresses  the  design  and  prototyping  of  a  six  degree-of-freedom  (DOF)  Cartesian  coordinate  hand 
controller  for  this  purpose.  The  device  design  recorrunended  is  an  XYZ  stage  attached  to  a  three-roll 
wrist  which  positions  flight-type  handgrip.  Six  degrees  of  freedom  are  transduced  and  control 
brushless  DC  motor  servo  electronics  similar  in  design  to  those  used  in  computer  controlled  robotic 
manipulators.  This  general  approach  supports  scaled  force,  velocity,  and  position  feedback  to  aid  an 
operator  in  achieving  telepresence.  The  generality  of  the  device  and  control  system  characteristics 
allow  the  use  of  inverse  dynamics  robotic  control  methodology  to  project  slave  robot  system  forces 
and  inertias  to  the  operator  (in  scaled  form)  and  at  the  same  time  to  reduce  the  apparent  inertia  of  the 
robotic  handcontroller  itself. 

The  handcontroller  unit  which  was  designed  and  simulated  in  the  initial  effort  uses  brushless  DC 
servo  motors,  integral  pancake  resolvers,  and  limited  gearing  to  produce  an  extremely  simple  and 
rugged  mechanical  design  requiring  minimal  routine  maintenance.  The  electronic  control  unit 
designed  is  a  compact  low  power  robotic  driver  which  supports  robotic  manipulator  systems  needing 
up  to  nominally  1500  W  of  power  from  a  personal  computer  sized  package.  The  current  control 
design,  which  is  not  multiple  fault  tolerant,  can  be  extended  in  this  area  to  make  flight  control  or 
space  use  possible. 

The  proposed  handcontroller  will  have  advantages  in  space-based  applications  where  an 
operator  must  control  several  robot  arms  in  a  simultaneous  and  coordinated  fashion.  It  will  also  have 
applications  in  intravehicular  activities  (within  the  Space  Station)  such  as  microgravity  experiments 
in  metallurgy  and  biological  experiments  that  require  isolation  from  the  astronauts’  environment.  For 
ground  applications,  the  handcontroller  will  be  useful  in  underwater  activities  where  the  generality 
of  the  proposed  handcontroller  becomes  an  asset  for  operation  of  many  different  manipulator  types. 
Also  applications  will  emerge  in  the  Military,  Construction,  and  Maintenance/Manufacturing 
areas  including  ordnance  handling,  mine  removal,  NBC  (Nuclear,  Chemical,  Biological)  operations, 
control  of  vehicles,  and  operating  strength  and  agility  enhanced  machines.  Future  avionics 
applications  including  advanced  helicopter  and  aircraft  control  may  also  become  important. 


1 .  The  work  reported  here  has  been  supported  by  NASA  Johnson  Space  Center  under  contract  NAS-9- 1 8094. 

2.  Poster  version. 
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ON  THE  ROLE  OF  EXCHANGE  OF  POWER  AND  INFORMATION 
SIGNALS  IN  CONTROL  AND  STABILITY  OF 
THE  HUMAN-ROBOT  INTERACTION 


H.  Kazerooni 
University  of  Minnesota 
Minneapolis,  Minnesota 


A  human’s  ability  to  perform  physical  tasks  is  limited,  not  only  by  his^  intelligence,  but  by  his 
physical  strength.  If,  in  an  appropriate  environment,  a  machine’s  mechanical  power  is  closely  inte¬ 
grated  with  a  human  ann’s  mechanical  power  under  the  control  of  the  human  intellect,  the  resulting 
system  will  be  superior  to  a  loosely  integrated  combination  of  a  human  and  a  fully  automated  robot. 
Therefore,  we  must  develop  a  fundamental  solution  to  the  problem  of  “extending”  human  mechani¬ 
cal  power.  The  work  presented  here  defines  “extenders”  as  a  class  of  robot  manipulators  worn  by 
humans  to  increase  human  mechanical  strength,  while  the  wearer’s  intellect  remains  the  central  con¬ 
trol  system  for  manipulating  the  extender.  The  human,  in  physical  contact  with  the  extender, 
exchanges  power  and  information  signals  with  the  extender. 

The  aim  is  to  determine  the  fundamental  building  blocks  of  an  intelligent  controller,  a  controller 
which  allows  interaction  between  humans  and  a  broad  class  of  computer-controlled  machines  via 
simultaneous  exchange  of  both  power  and  information  signals.  Traditionally,  human  interaction 
with  active  systems  (self-powered  machines)  has  been  defined  as  the  exchange  of  “information 
signals”  only.  For  example,  the  human  sends  information  signals  to  an  electric  mixer  (an  active 
system)  by  pushing  a  “start”  button,  but  the  human  does  not  transfer  power  to  the  mixer  and  does  not 
feel  the  actual  load  on  the  mixer  blades.  As  in  this  example,  the  prev^ent  trend  in  automation  has 
been  to  physically  separated  the  human  from  the  machine  so  the  human  must  always  send 
information  signals  via  an  intermediary  device  (e.g.,  joystick,  pushbutton,  lightswitch).  Extenders, 
however  are  perfect  examples  of  self-powered  machines  that  are  built  and  controlled  for  the  optimal 
exchange  of  power  and  information  signals  with  humans.  The  human  wearing  the  extender  is  in 
physical  contact  with  the  machine,  so  power  transfer  is  unavoidable  and  information  signals  from  the 
human  help  to  control  the  machine.  Commands  are  transferred  to  the  extender  via  the  contact  forces 
and  the  EMG  signals  between  the  wearer  and  the  extender,  thus  eliminating  the  need  for  a  Joystick, 
pushbutton,  lightswitch  or  keyboard.  This  allows  the  person  to  maneuver  his  hand  more  naturally.  If 
“talking”  is  defined  as  a  natural  method  of  communication  between  two  people,  then  we  would  like 
to  communicate  with  a  computer  naturally  by  talking  rather  than  by  using  a  keyboard.  The  same  is 
true  here:  if  we  define  “maneuvering  the  hand”  as  a  natural  method  of  moving  loads,  then  we  would 
like  to  move  only  the  hand  to  maneuver  a  load,  as  opposed  to  using  a  keyboard  or  joystick.  The 
extender  augments  human  motor  ability  without  accepting  any  explicit  commands:  it  accepts  the 
EMG  signals  and  the  contact  force  between  the  person’s  arm  and  the  extender,  and  the  extender 
“translates”  them  into  a  desired  position.  In  this  unique  configuration,  mechanical  power  transfer 
between  the  human  and  the  extender  occurs  because  the  human  is  pushing  against  the  extender.  The 
extender  transfers  to  the  human’s  hand,  in  feedback  fashion,  a  scaled-down  version  of  the  actual 


1 .  The  pronouns  “he,”  “his,”  and  “him”  used  in  this  abstract  are  not  meant  to  be  gender-specific. 
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external  load  which  the  extender  is  manipulating.  This  natural  feedback  force  on  the  human’s  hand 
allows  him  to  “feel”  a  modified  version  of  the  external  forces  on  the  extender. 

Further  clarification  of  the  concept  of  transfer  of  power  and  information  signals  follows.  The 
information  signals  from  the  human  (e.g.,  EMG  sign^s)  to  the  computer  reflect  human  cognitive 
ability,  and  the  power  transfer  between  the  human  and  the  machine  (e.g.,  physical  interaction) 
reflects  human  physical  ability.  Thus  the  information  transfer  to  the  machine  augments  cognitive 
ability,  and  the  power  transfer  augments  motor  ability.  These  two  actions  are  coupled  through  the 
human  cognitive/motor  dynamic  behavior.  The  goal  is  to  derive  the  control  rules  for  a  class  of  com¬ 
puter-controlled  machines  that  augment  human  physical  and  cognitive  abilities  in  certain  manipula¬ 
tive  tasks. 
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A  TWO  DIMENSIONAL  ARTIFICIAL  REALITY 


Myron  W.  Krueger 
Artificial  Reality 
Vernon,  Connecticut 


The  current  presumption  is  that  is  necessary  to  don  goggles,  gloves  and  a  data  suit  to  experience 
artificial  reality.  However,  there  is  another  technology  that  offers  an  alternative  or  complement  to  the 
encumbering  techniques  associated  with  NASA.  In  VIDEOPLACE,  your  image  appears  in  a  2D 
graphic  world  created  by  a  computer.  The  VIDEOPLACE  computer  can  analyze  a  person’s  image  in 
1/30  second  and  can  detect  when  an  object  is  touched  (Fig.  1).  Thus,  it  can  generate  a  graphic  or 
auditory  response  to  an  illusory  contact. 

VIDEOPLACE  technology  exists  in  two  formats:  the  VIDEODESK  and  the  VIDEOPLACE.  In 
the  VIDEODESK  environment,  the  image  of  your  hands  can  be  used  to  perform  the  normal  mouse 
functions,  such  as  menuing  and  drawing  (Fig.  2).  In  addition,  you  have  the  advantage  of  multippoint 
control.  For  instance,  you  can  use  the  thumbs  and  forefingers  of  each  hand  as  control  points  for  a 
spline  curve  (Fig.  3).  Perhaps  most  important,  the  image  of  your  hands  can  be  compressed  and 
transmitted  to  a  colleague  over  an  ISDN  voice  channel  to  appear  on  the  remote  screen  superimposed 
over  identical  information.  Likewise,  the  image  of  your  colleague’s  hands  can  appear  on  both 
screens.  The  result  is  that  the  two  of  you  can  use  your  hands  to  point  to  features  on  your  respective 
screens  as  you  speak,  exactly  as  you  would  if  you  were  sitting  together  (Fig.  4). 

In  the  VIDEOPLACE  environment,  you  can  interact  with  graphic  creatures  and  the  images  of 
other  people  in  other  locations  in  a  graphic  world.  Your  whole  body  can  be  moved,  scaled  and 
rotated  in  real-time  without  regard  to  the  laws  of  physics.  Thus,  VIDEOPLACE  can  be  used  to  create 
a  fantasy  world  in  which  the  laws  of  cause  and  effect  are  composed  by  an  artist  (Fig.  5)  (Fig.  6). 
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Figure  1.  Video  touch. 


Figure  3.  Multipoint  control. 
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Figure  4.  Teleconununication. 


Figure  5.  Landing  in  an  artificial  reality. 


Figure  6.  Parent  and  child  size  reversal. 
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BINAUR4L  ROOM  SIMULATION 


H.  Lehnert,  J.  Blauert,  and  W.  Pompetzki 
Ruhr  Universitat  Bochum 
Bochum,  FR  GERMANY 


In  every-day  listening  the  auditory  event  perceived  by  a  listener  is  determined  not  only  by  the 
sound  signal  that  a  sound  emits  but  also  by  a  variety  of  environmental  parameters.  These  parameters 
are  the  position,  orientation  and  directional  characteristics  of  the  sound  source,  the  listener’s  position 
and  orientation,  the  geometrical  and  acoustical  properties  of  surfaces  which  affect  the  sound  field 
and  the  sound  propagation  properties  of  the  surrounding  fluid.  A  complete  set  of  these  parameters 
can  be  called  an  Acoustic  Environment.  If  the  auditory  event  perceived  by  a  listener  is  manipulated 
in  such  a  way  that  the  listener  is  shifted  acoustically  into  a  different  acoustic  environment  without 
moving  himself  physically,  a  Virtual  Acoustic  Environment  has  been  created.  Here,  we  deal  with  a 
special  technique  to  set  up  nearly  arbitrary  Virtual  Acoustic  Environments,  the  Binaural  Room 
Simulation.  The  purpose  of  the  Binaural  Room  Simulation  is  to  compute  the  binaural  impulse 
response  related  to  a  virtual  acoustic  environment  taking  into  account  all  parameters  mentioned 
above. 

One  possible  way  to  describe  a  Virtual  Acoustic  Environment  is  the  concept  of  the  virtual  sound 
sources.  Each  of  the  virtual  sources  emits  a  certain  signal  which  is  correlated  but  not  necessarily 
identical  with  the  signal  emitted  by  the  direct  sound  source.  If  source  and  receiver  are  non  moving, 
the  acoustic  environment  becomes  a  linear  time-invariant  system.  Then,  the  Binaural  Impulse 
Response  from  the  source  to  a  listener’s  eardrums  contains  all  relevant  auditory  information  related 
to  the  Virtual  Acoustic  Environment.  Listening  into  the  simulated  environment  can  easily  be 
achieved  by  convolving  the  Binaural  Impulse  Response  with  diy  ;ignils  and  rcp.cr.'nting  the  results 
via  headphones. 

Our  simulation  system  contains  the  following  steps.  First  the  parameters  of  the  environment  to  be 
simulated  are  entered  into  the  computer  using  a  CAD  program  geometric  data  of  the  room  and 
databases  for  acoustic  data  of  the  walls  and  the  directional  characteristics.  Position  and  orientation  of 
source  and  receiver  are  defined  by  local  coordinate  systems.  Then,  the  positions  and  orientations  of 
the  virtual  sound  sources  are  computed  by  ray  tracing  or  mirror  imaging  algorithms  (refs.  1,  2, 
and  3).  The  signals  of  the  virtual  sources  are  derived  by  convoluting  all  impulse  responses  of  the 
walls  involved  in  the  sound  path  of  the  virtual  source.  The  binaural  room  impulse  response  is  then 
computed  by  propagating  the  signal  of  each  virtual  source  to  the  listener  and  convolving  the  result 
with  the  external-ear  impulse  response  of  the  listener  for  the  respective  direction  of  incidence. 
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ACTIVE  LOCALIZATION  OF  VIRTUAL  SOUNDS 


J.  M.  Loomis,  C.  Hebert,  and  J.  G.  Cicinelli 
University  of  California 
Santa  Barbara,  California 


We  describe  a  virtual  sound  display  built  around  a  12  MHz  80286  microcomputer  and  special- 
purpose  analog  hardware.  The  display  implements  most  of  the  primary  cues  for  sound  localization  in 
the  ear-level  plane.  Static  information  about  direction  is  conveyed  by  interaural  time  differences  and, 
for  frequencies  above  1800  Hz,  by  head  .sound  shadow  (interaural  intensity  differences)  and  pinna 
sound  shadow.  Static  information  about  distance  is  conveyed  by  variation  in  sound  pressure  (first 
power  law)  for  all  frequencies,  by  additional  attenuation  in  the  higher  frequencies  (simulating  atmo¬ 
spheric  absorption),  and  by  the  proportion  of  direct  to  reverberant  sound.  When  the  user  actively 
locomotes,  the  changing  angular  position  of  the  .source  occasioned  by  head  rotations  provides  further 
information  about  direction  and  the  changing  angular  velocity  produced  by  head  translations  (motion 
parallax)  provides  further  information  about  distance.  Judging  both  from  informal  observations  by 
users  and  from  objective  data  obtained  in  an  experiment  on  homing  to  virtual  and  real  sounds,  we 
conclude  that  simple  displays  such  as  this  are  effective  in  creating  the  perception  of  external  sounds 
to  which  subjects  can  home  with  accuracy  and  ease. 
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HUMAN-MACHINE  INTERFACE  HARDWARE:  THE  NEXT  DECADE 


E.  A.  Marcus 
EXOS,  Inc. 

Lexington,  Massachusetts 

In  order  to  understand  where  human-machine  interface  hardware  is  headed,  it  is  important  to 
understand  where  we  are  today,  how  we  got  there,  and  what  our  goals  for  the  future  are. 

As  computers  become  more  capable,  faster,  and  programs  become  more  sophisticated,  it 
becomes  apparent  that  the  interface  hardware  is  the  key  to  an  exciting  future  in  computing.  One  only 
has  to  look  at  CAD  workstation  hardware  and  the  array  of  tablet,  stylus,  trackball,  knob,  and  mouse 
devices  which  are  combined  with  a  maze  of  menus  and  function  keys  to  understand  the  issues.  These 
programs  can  create  visual  representations  in  2  or  3-D,  illustrate  attributes  of  the  object,  such  as 
stress  and  temperature  distributions,  or  model  and  animate  the  object  as  it  interacts  with  other 
objects.  How  can  a  user  interact  and  control  this  seemingly  limitless  array  of  parameters  effectively? 
Today,  the  answer  is  most  often  a  limitless  array  of  controls.  The  link  between  these  controls  and 
human  sensory  motor  capabilities  does  not  utilize  existing  human  capabilities  to  their  full  extent. 

Imagine  if  the  developers  of  the  first  car  had  electronic  controls  at  their  disposal  and  decided  to 
control  all  car  functions  through  a  tablet  or  an  array  of  switches  or  dials.  They  might  have  argued 
against  a  steering  wheel  as  follows,  “If  a  steering  wheel  is  used,  both  hands  are  completely  occupied 
at  all  time.  How  would  you  control  the  speed,  or  any  of  the  other  functions?”  The  rebuttal  might 
have  been,  “Use  foot  pedals  which  is  natural  due  to  experience  with  walking  or  cycling.”  The  tablet 
advocate  would  have  said,  “No  one  will  want  to  use  their  feet,  it’s  just  not  natural.  Even  if  people 
would  try  it,  it  wouldn’t  be  smooth  and  it  would  damage  the  whole  concept  of  the  car.”  As  you  can 
see,  if  cars  evolved  this  way,  they  would  be  a  maze  of  dials,  menus  and  controls  which  saturate 
human  sensory  capabilities.  We’d  probably  have  speed  limits  of  30  mph  due  to  the  limitations  in  the 
reaction  time  as  set  by  ine  interface  hardware.  Luckily,  the  car  evolved  to  take  more  advantage  of 
human  capabilities. 

To  use  the  car  analogy  again,  when  the  car  was  bom,  they  didn’t  ask  what  a  car  in  the  year  2000 
would  look  like  and  leave  “hooks”  and  “scars”  for  improved  technology.  If  they  had,  they  would 
have  been  wrong,  as  few  if  any  people  are  able  to  anticipate  technology  breakthroughs.  They  did, 
however,  provide  the  best  that  was  possible  at  the  time  and  got  the  product  out  to  as  many  people  as 
possible  and  let  time,  experience,  market  demand,  and  technology  innovation  lead  to  the  future. 

Interface  hardware  for  teleoperation  and  virtual  environments  is  now  facing  just  such  a  crossroad 
in  design.  Therefore,  we  as  developers  need  to  explore  how  the  combination  of  interface  hardware, 
human  capabilities,  and  user  experience  can  be  blended  to  get  the  best  performance  today  and  in  the 
future. 
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COORDINATION  OF  HUMAN  UPPER  ARM  AND  FOREARM  MOTION 

IN  RAPID  EXTENSIONS 


Mahmood  Nahvi 

California  Polytechnic  State  University 
San  Luis  Obispo,  California 


In  many  movements  of  the  upper  limb  such  as  reaching,  positioning,  and  diplacing  the  objects, 
the  hand  moves  smoothly  along  a  uni-directional  planar  trajectory  with  a  bell-shaped  speed  profile. 

Because  of  variability  of  the  load,  gravitational  and  velocity  interaction  forces  between  its  seg¬ 
ments,  the  dynamics  of  the  arm  during  the  motion  is  very  complex.  Motion  of  the  upper  arm  and  the 
forearm  are  coordinated  to  produce  smooth  movements  despite  such  complex  dynamics.  This  coor¬ 
dination  constitutes  an  important  organizational  feature  of  arm  movement.  The  present  paper 
describes  some  experimental  results  related  to  the  above  and  interprets  their  role  in  producing 
smooth  motion  of  the  hand. 

Trajectories  of  the  right  upper  limb  in  vertical  plane  and  the  simultaneous  muscles’  EMG  activi¬ 
ties  were  recorded  for  extensions  of  various  amplitudes  under  four  loads.  The  forearm  trajectories 
are  smooth  with  bell-shaped  speed  profiles.  The  upper  arm  trajectories  may  have  bimodal  speed 
profiles  and  three  segments,  resembling  an  inverted  “Z.”  Segmentation  is  sharper  for  the  points  near 
the  shoulder  joint,  and  is  accentuated  by  load.  As  one  moves  from  the  central  points  near  the  shoul¬ 
der  joint  to  the  periprieral  points  near  the  hand,  the  three  segments  merge  and  result  in  smooth  curves 
with  single-peaked  speed  profiles.  The  three  segments  have  a  coherent  time  course  and  can  be  iden¬ 
tified  rather  accurately  and  non-ambiguously. 

The  observed  trajectories  and  the  EMG  patterns  reveal  an  effective  coordination  strategy  which 
utilizes  the  structure  and  the  dynamics  of  the  moving  arm  to  produce  smooth  movement. 
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MANUAL  DISCRIMINATION  OF  FORCE 


X.  D.  Pang,  H.  Z.  Tan,  and  N.  I.  Durlach 
Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts 


Optimal  design  of  human-machine  interfaces  for  teleoperators  and  virtual-environment  systems 
which  involve  the  tactual  and  kinesthetic  modalities  requires  knowledge  of  the  human’s  resolving 
power  in  these  modalities.  The  resolution  of  the  interface  should  be  appropriately  matched  to  that  of 
the  human  operator.  We  report  some  preliminary  results  on  the  ability  of  the  human  hand  to  distin¬ 
guish  small  differences  in  force  under  a  variety  of  conditions.  Experiments  were  conducted  on  force 
discrimination  with  the  thumb  pushing  an  interface  that  exerts  a  constant  force  over  the  pushing  dis¬ 
tance  and  the  index  finger  pressing  against  a  fixed  support.  The  dependence  of  the  sensitivity  index 
d'  on  force  increment  can  be  fit  by  a  straight  line  through  the  origin  and  the  just-noticeable  differ¬ 
ence  (JND)  in  force  can  thus  be  described  by  the  inverse  of  the  slope  of  this  line.  The  receiver-oper¬ 
ating-characteristic  (ROC)  was  measured  by  varying  the  a  priori  probabilities  of  the  two  alternatives, 
reference  force  and  reference  force  plus  an  increment,  in  one-interval,  two-alternative,  forced-choice 
experiments.  When  plotted  on  normal  deviate  coordinates,  the  ROCs  were  roughly  straight  lines  of 
unit  slope,  thus  supporting  the  assumption  of  equal-variance  normal  distributions  and  the  use  of  the 
conventional  d’  measure.  The  JND  was  roughly  6-8%  for  reference  force  ranging  from  2.5  to 
10  newtons,  pushing  distance  from  5  to  30  mm,  and  initial  finger-span  from  45  to  125  mm.  Also,  the 
JND  remained  the  same  when  the  subjects  were  instructed  to  change  the  average  speed  of  pushing 
from  23  to  153  mm/sec.  The  pushing  was  terminated  by  reaching  either  a  wall  or  a  well,  and  the 
JNDs  were  essentially  the  same  in  both  cases. 
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IMPLEMENTATION  OF  A  THREE  DEGREE  OF  FREEDOM, 
MOTOR/BRAKE  HYBRID  FORCE  OUTPUT  DEVICE 
FOR  VIRTUAL  ENVIRONMENT  CONTROL  TASKS 


Massimo  Russo,  Alfred  Tadros,  Professor  Woodie  Flowers,  and  Professor  David  Zelter 

Massachusetts  Institute  of  Technology 
Cambridge,  Massachusetts 


The  advent  of  high  resolution,  physical  model  based  computer  graphics  has  left  a  gap  in  the 
design  of  input/output  technology  appropriate  for  interacting  with  such  complex  virtual  world  mod¬ 
els.  Since  virtual  worlds  consist  of  physical  models,  it  is  appropriate  to  output  the  inherent  force 
information  necessary  for  the  simulation  to  the  user.  The  detailed  design,  construction,  and  control  of 
a  three  degree  freedom  force  output  joystick  will  be  presented. 

A  novel  kinematic  design  allows  all  three  axes  to  be  uncoupled,  so  that  the  system  inertia  matrix 
is  diagonal.  The  two  planar  axes  are  actuated  through  an  offset  gimbal,  and  the  third  through  a 
sleeved  cable.  To  compensate  for  friction  and  inertia  effects,  this  transmission  is  controlled  by  a 
force  feedforward  and  a  closed  force  feedback  proportional  loop.  Workspace  volume  is  a  cone  of 
512  cubic  inches,  and  the  device  bandwidth  is  maximized  at  60  Hz  for  the  two  planar  and  30  Hz  for 
the  third  axis.  Each  axis  is  controlled  by  a  motor/proportional  magnetic  particle  brake  combination 
fixed  to  the  base.  The  innovative  use  of  motors  and  brakes  allows  objects  with  high  resistive  torque 
requirements  to  be  simulated  without  the  stability  and  related  safety  issues  involved  with  high 
torque,  energy  storing  motors  alone.  Position,  velocity,  and  applied  endpoint  force  are  sensed 
directly.  Different  control  strategies  are  discussed  and  implemented,  with  an  emphasis  on  how  vir¬ 
tual  environment  force  information,  generated  by  the  MIT  Media  Lab  Computer  Graphics  and  Ani¬ 
mation  Group  BOLIO  system,  is  transmitted  to  the  device  controller. 

The  design  constraints  for  a  kinesthetic  force  feedback  device  can  be  summarized  as: 

How  can  the  symbiosis  between  the  sense  of  presence  in  the  virtual  environment  be  maximized 
without  compromising  the  interaction  task,  under  the  constraints  of  the  mechanical  device 
limitations? 

Research  in  this  field  will  yield  insights  to  the  optimal  human  sensory  feedback  mix  for  a  wide 
spectrum  of  control  and  interaction  problems.  A  flexible  research  tool  that  is  designed  as  an  easily 
reproducible  product  prototype  has  been  constructed  to  explore  the  variety  of  possible  force 
interaction. 


Introduction 


The  advent  of  high  resolution,  physical  model  based  computer  graphics  has  left  a  gap  in  the 
design  of  input/output  technology  appropriate  for  interacting  with  such  complex  virtual  world  mod¬ 
els.  Visual  display  technology  has  improved  dramatically,  driven  by  developing  communication 
through  the  most  information  rich  sensory  channel.  Advances  in  stereographic  displays  and  more 
proprioceptive  input  devices  that  contain  higher  degreee  of  freedom  position  information,  such  as 
full  hand  or  body  gesture  recognition,  still  rely  only  on  the  visual  sense.  The  user  operates  in  a  world 
of  relative  paralasys,  not  able  to  feel,  grasp,  touch,  smell,  or  poke  the  objects  that  she  interacts  with. 

Since  virtual  worlds  are  based  on  physical  models,  it  is  appropriate  to  output  the  inherent  force 
information  necessary  for  the  simulation  to  the  user.  Before  one  commits  large  resources  to  the  con¬ 
struction  of  such  force  output  machines,  the  appropriate  design  constraints  and  problem  definitions 
for  simulated  environment  interactions  must  be  addressed.  The  problem  can  be  summarised  as: 

How  can  the  symbiosis  between  the  sense  of  presence  in  the  virtual  environment  be  maximised 
without  compromising  the  interaction  task,  under  the  constraints  of  the  mechanical  device 
limitations? 

Research  in  this  field  will  yield  insights  to  the  optimal  human  sensory  feedback  mix  for  a  wide 
spectrum  of  control  and  interaction  problems:  telepresence  and  supervisory  control  in  remote 
manipulator  tasks,  advanced  computer  assisted  engineering  design  tools,  and  surgical  or  flight  simu¬ 
lators  among  many  others.  A  flexible  research  tool  to  explore  the  variety  of  possible  force  interac¬ 
tions  and  is  nevertheless  designed  as  an  easily  reproducable  product  prototype  has  been  constructed. 

Overview  and  Design  Specification 

Traditional  force  output  devices  have  concentrated  on  large,  global  movements,  motivated  by  the 
scale  of  the  teleoperated  robot  motion.  In  general,  an  analog  slave  sensor  feedback  signal  servos  the 
master  Joint  motors.  The  geometric  configuration  and  physical  dimensions  of  the  controlled  manipu¬ 
lator  and  of  the  master  are  the  same.  In  many  cases,  however,  geometric  equality  is  not  necessary 
and  can  even  hinder  a  particular  successful  task  completion.  Recently,  in  the  flexible,  unspecified 
tasks  that  are  possible  with  virtual  environments,  the  goal  is  to  design  the  force  output  device  so  that 
its  own  dynamics  are  transparent  and  yet  also  can  simulate  as  wide  a  frequency  band  of  forces  as 
possible.  The  range  of  motion  used  in  interacting  with  a  computer  can  be  limited  to  a  small  surface 
or  volume,  one  need  only  think  of  the  small  “mouse”  interface  device  movement.  The  design  of  a 
force  output  device  for  virtual  environments  should  maximize  the  device  fidelity  under  the  general 
constraints  of  size,  cost,  limited  computation  speed,  and  safety.  These  are  the  general  constraints  that 
drove  the  design  of  the  presented  force  feedback  interface.  The  design  catagories  are  detailed  below. 
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Design  Categories: 


Engineering  Development: 

Design  Simplicity: 

Appearance 

Limited  Size 

Robust 

Versatile 

Modular 

Expandable 

Safety 

Difficulty  of  implementation 
Technological  base 
Cost 

Controllability: 

Interface  Transparency: 

Can  be  counterbalanced  (Electronically  or  Mechanically)  Backdrivable 
High  DOF 
Uncoupled  axes 
Low  degree  of  computation: 

Full  State  measurement 

Integrated  into  system  w/o  control/graphic  display  interference 
Limited  control  logic 
High  Position  Resolution 
Uncoupled  axes 
Closed  or  open  loop  control: 

FuU  State  measurement 
Stability: 

Stiff 

High  natural  frequency 
Human-Device  Interaction: 

Secondary  Function  Control 

Non-fatiguing 

Safety 

Human  arm  limitations  never  exceeded 
Interface  Transparency 
Variable  force  feedback  ratio 

The  described  force  output  joystick  incorporates  linear  but  no  rotational  force  output.  Many 
manual  tasks  in  which  force  information  is  of  utmost  importance  and  that  are  completed  with  the  use 
of  tools  can  be  limited  to  three  linear  degrees  of  freedom.  The  metaphor  used  as  a  design  guideline, 
which  allows  for  a  tractable  problem  within  the  design  requirements,  is  a  tool  endpoint  force  simula¬ 
tor.  To  maintain  design  simplicity  and  limit  the  size  without  compromising  device  versatility,  the 
joystick  endpoint  is  limited  to  the  three  degrees  of  linear  motion.  It  can  be  upgraded  to  include  a 
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small  three  degrees  of  freedom  torque  ijutput  mechanism,  so  a  full  6  degrees  of  freedom  force  and 
torque  output  could  be  realised. 


i\.\j  point  c?utact  can  by  mndul'-ted  according  nn  the  chr..  dcter;.>u(.3  imposed  by  the  simulatioii. 
The  joystick  endpoint  is  a  single  probe  with  which  to  explore  the  virtual  computer  space.  How 
effective  such  an  approach  is  in  transmitting  tactile  information  is  a  matter  of  ongoing  research 
(Minsky).  Full,  high  resolution  tactile  displays  have  a  large  number  of  independent  degrees  of  free¬ 
dom,  determined  by  the  size  and  resolution  of  the  display.  Each  tactile  “pixel”  must  be  indepen¬ 
dently  controlled,  so  devices  of  this  kind  are  difficult  to  implement.  Examples  include  Braille 
machines  for  the  blind,  which  are  the  state  of  the  art  for  such  devices,  and  it  is  an  open  question  if 
their  resolution  is  sufficient  for  effective,  more  general  representation  tactile  displays. 


A  fine  tool  is  not  grasped  with  the  whole  hand,  but  rather  like  a  pen  between  one’s  fingers.  The 
force  magnitude  in  such  and  interaction  is  much  lower  than  in  full  arm  movement.  The  restriction  to 
finer  forces  and  motions,  which  is  appropriate  for  the  limited  workspace  requirement  of  the  human 
arm  and  hand,  allows  the  Joystick  to  be  constrained  in  size  since  the  actuator  brakes  and  motors  can 
have  lower  torque  ouputs.  Also,  the  smaller  motors  more  easily  meet  the  safety  requirements  and  the 
limitation  to  three  degrees  of  freedom  decreases  the  difficulty  of  implementation  and  cost.  The  three 
degrees  of  freedom  can  be  chosen  such  that  the  axes  are  perpendicular  and  all  the  actuators  are 
attached  to  ground,  thus  eliminating  cross  coupling  between  the  endpoint  principal  directions  and  the 
actuator  torques.  Rotary,  no  torque,  ripple,  low  inertia  motors  and  proportional  magnetic  particle 
brakes  with  a  simple  cable  transmission  for  the  third  axis,  and  most  of  the  sensors  are  readily  avail¬ 
able  from  existing  technology.  Only  a  three  axis  linear  load  cell  of  sufficiently  small  size  and  mass 
had  to  be  specifically  designed  and  constructed. 


Control  Requirements 

The  detailed  mechanical  design  of  the  joystick  is  dominated  by  the  control  requirements  of  the 
simulation.  To  both  impose  the  arbitrary  simulation  dynamics  onto  the  joystick  endpoint  and  mini¬ 
mize  user  fatigue  when  the  joystick  is  used  as  a  simple  position  input  device,  the  actual  characteristic 
or  plant  dynamics  should  not  interfere  with  the  simulation.  This  interface  transparency  is  from  the 
perspective  of  the  user  interacting  with  the  joystick.  One  input  to  the  plant  is  the  force  exerted  by  the 
user  on  the  endpoint.  The  simulation,  however,  controls  the  desired  dynamics  through  the  input  sig¬ 
nals  to  the  motors  and  brakes.  The  control  strategy  needs  to  maintain  the  simulation  dynamics  under 
varying  user  force  inputs,  or  under  the  changing  coupling  characteristics  between  the  user’s  arm  and 
plant  dynamics.  The  simulation  essentially  modulates  the  brake  and  motor  signals  so  as  to  match  the 
endpoint  impedance  or  effective  mass,  stiffness,  and  damping  characteristics  to  that  of  the  virtual 
dynamics.  The  mechanical  design  maximizes  the  inherent  device  transparency  and  so  minimized  the 
control  effort  needed  to  backdrive  the  joystick.  It  has  a  bandwidty  of  60  Hz  for  the  gimbal  driven 
axes  and  30  Hz  for  the  cable  drive.  Because  of  closed  loop  force  control  stability  issues,  open  loop 
backdriveability  is  obtained  wherever  possible.  This  necessitates  a  direct  drive  between  the  brakes, 
motors,  and  joystick  endpoint  to  minimize  stiction,  viscous  friction  and  effective  actuator  inertia, 
which  are  all  amplified  by  a  transmission  ratio.  Direct  drive  in  turn  requires  large  motors  for  a  signif¬ 
icant  torque.  All  the  motors  and  brakes  are  fixed  to  ground  and  thus  their  weight  and  reaction  torques 
need  not  be  compensated  for.  Because  of  the  much  higher  torque  output  of  the  brake  for  a  given  size 
and  power  input,  any  high  torque  requirements  of  the  simulation  will  be  met  by  the  brake,  an 
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inherently  passive,  no  energy  storing  and  therefore  safe  torque  device.  Thus,  both  backdriveability 
and  output  force  are  maximized  within  the  size  and  safety  constraints. 

No  known  res»»^»rr'h  bas  been  conducted  in  the  apnropriate  strategy  for  hybrid  independent  pro¬ 
portional  brake  and  motor  control.  This  novel  combination  promises  to  yield  new  methods  of 
increasing  stability  and  safety  bound  in  actively  controlled  mechanisms  without  decreasing  the 
closed  loop  device  stiffness.  The  control  of  brakes  necessitates  a  direct  measurement  of  the  contact 
force  between  the  endpoint  and  user.  Both  for  this  reason  and  to  actively  backdrive  the  R-axis 
transmission,  a  loadcell  3-axis  force  sensor  is  implemented  in  the  device. 

The  actual  joystick  endpoint  position,  velocity,  and  force  are  necessary  to  determine  the  charac¬ 
teristics  of  the  interface  dynamics.  The  servo  loop  sampling  frequency  is  limited  by  the  bandwidth  of 
the  simulated  environment.  As  all  the  joystick  control  must  operate  in  real  time,  the  degree  of  com¬ 
putation  lag  in  the  control  loop  must  be  minimized.  To  achieve  this  and  to  eliminate  quantization  and 
noise  errors  at  low  velocities  introduced  by  digital  differentiation,  both  motor/brake  shaft  position 
and  velocity  are  measured  directly.  The  endpoint  position  and  velocity  can  then  be  found  through  a 
simple  matrix  position  and  Jacobian  transformation.  The  joystick  linkage  is  chosen  such  that  all 
actuator  induced  endpoint  velocities  are  perpendicular.  The  axes  are  therefore  uncoupled  and  any 
actuator  exerts  a  force  in  one  of  the  endpoint  principle  directions.  The  system  inertia  matrix  is  thus 
only  diagonal,  though  time  variant. 

Future  Work 

Some  simple  texture  and  object  simulation  code  has  been  implemented  successfully.  Present 
research  efforts  are  concentrated  on  expanding  the  complexity  of  a  real-time  virtual  world  interaction 
by  utilising  the  physically  based  graphics  animation  package  in  the  MIT  Media  Lab  Computer 
Graphics  and  Animation  Group.  Various  studies  of  the  effect  of  multiple  sensory  interaction  on  real¬ 
ism  in  simulations  and  the  associated  mechanical  device  complexity  cost  are  proposed,  with  the  pre¬ 
sent  device  as  a  research  tool  to  quantify  the  effect  of  force  information  in  successful  task  comple¬ 
tion  and  enhanced  apparent  realism. 
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HUMAN  FACTORS  REQUIREMENTS  FOR  TELEROBITIC  COMMAND 
AND  CONTROL:  THE  EUROPEAN  SPACE  AGENCY  EXPERIMENTAL 

PROGRAMME 

Robert  J.  Stone 

The  National  Advanced  Robotics  Research  Centre 
Salford,  United  Kingdom 


Space  Telerobotics  research,  performed  under  contract  to  the  European  Space  Agency  (ESA), 
concerning  the  execution  of  human  factors  experiments,  and  ultimately  leading  to  the  development 
of  a  telerobotics  test  bed,  has  been  carried  out  since  1985  by  a  British  Consortium  consisting  of 
British  Aerospace,  the  United  Kingdom  Atomic  Energy  Authority  and,  more  recently,  the  UK 
National  Advanced  Robotics  Research  Centre.  The  principal  aim  of  the  first  study  of  the  series  was 
to  derive  preliminary  requirements  for  a  teleoperation  servicing  system,  with  reference  to  two  mis¬ 
sion  model  scenarios.  The  first  scenario  introduced  the  problem  of  communications  time  delays,  and 
their  likely  effect  on  the  ground-based  operator  in  control  of  a  manipulator  system  on  board  an 
unmanned  servicing  vehicle  in  Low  Earth  Orbit.  In  the  second  scenario,  the  operator  was  located  on 
the  NASA  Orbiter  aft  flight  deck,  supervising  the  control  of  a  prototype  manipulator  in  the 
“servicing”  of  an  experimental  payload  in  the  cargo  bay  area.  Human  factors  analyses  centred  on 
defining  the  requirements  for  the  teleoperator  workstation,  such  as  identifying  basic  ergonomic 
requirements  for  workstation  and  panel  layouts,  defining  teleoperation  strategies,  developing 
alphanumeric  and  graphic  screen  formats  for  the  supervision  or  direct  control  of  the  manipulator,  and 
the  potential  applications  of  expert  system  technology. 

The  second  study  for  ESA  involved  an  experimental  appraisal  of  some  of  the  important  issues 
highlighted  in  the  first  study,  for  which  relevant  human  factors  data  did  not  exist.  Of  central  impor¬ 
tance  during  the  second  study  was  the  issue  of  connnunications  time  delays  and  their  effect  on  the 
manual  control  of  a  teleoperated  manipulator  from  a  ground-based  command  and  control  station.  An 
experimental  test  bed,  based  around  ASEA  Irb6  and  60  robots  was  developed,  incorporating  a  gen¬ 
eral  purpose  robot  controller,  a  reconfigurable  operator  workstation  and  a  modular  task  box.  The 
programme  was  divided  into  a  series  of  short  pilot  studies  and  two  principal  experiments.  The  aim  of 
the  pilot  studies  was  to  optimise  the  characteristics  of  hand  controllers  and  visual  feedback  condi¬ 
tions  prior  to  their  more  formal  evaluation  and  comparison  in  the  later  experiments.  The  experiments 
were  conducted  using  standard  psychological  paradigms,  involving  human  performance  comparisons 
over  a  range  of  3  and  6  degree-of-freedom  hand  controllers,  with  TV  and  graphics  displays  in  vari¬ 
ous  configurations.  Although  a  rigorous  experimental  approach  was  adopted  during  Teleoperation 
and  Control  Study  2,  it  was  evident  that  this  approach  suffers  a  number  of  limitations.  The  most 
critical  concerns  the  extent  to  which  the  data  and  conclusions  from  this  study  can  be  generalised  to 
future  scenarios.  Also,  it  is  recognised  that  the  performance  measures  of  time  and  accuracy  alone 
were  not  satisfactory,  as  they  each  provided  only  an  overall  index  of  performance  for  a  given  trial, 
not  an  analytical  index  of  performance  within  thf*  trial.  Clearly,  more  analytical  techniques  to  analyse 
operator  performance  in  detail  are  required,  together  with  a  methodological  framework  in  which  all 
techniques  can  be  used  to  judge  the  effect  of  “cost”  of  a  particular  technology,  not  only  on  human 
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teleoperation  performance,  but  also  on  the  total  telerobotic  system,  with  the  future  aim  of  defining 
operational  procedures. 


Study  3  ::  investigating  a  new  te*.Iiniquc  -  console  or  performance 


“logging”-  for  the  measurement  of  teleoperation  performance  to  complement  those  measures  used  in 
the  second  study,  such  as  primitive  and  whole-task  timings,  handling  errors,  debriefing  and  subjec¬ 
tive  workload  records  vNASA  TLX),  video  and  audio  recordings.  Instrumenting  the  human-system 
interface  to  record  and  store  both  multiple  gross  actions  and  much  finer  keystroke-level  parameters 
should  provide  a  more  detailed  insight  into  me  incidence,  cause  and  effect  of  interactions  between 
various  control  activities,  and  assist  the  experimenters  in  analyzing  those  control  behaviours  of  too 
fine  a  nature  to  be  extracted  from  video  records  alone  (such  as  fine  individu?’  joystick  motions). 
Such  a  keystroke  level  of  analysis  should  provide  data  suitable  for  future  performance  modelling 
activities  in  the  area  of  telerobotic  command  and  control. 


Included  among  the  parameters  to  be  recorded  in  Study  3  are  all  robot  control  joystick  axes,  all 
additional  joystick-mounted  controls  (where  used),  all  panel-mounted  switches,  all  camera  joystick 
controls,  all  direct  voice  input  utterances  (inciaence  of  onset  only),  all  robot  motions,  as  resolved  at 
the  end  effector  (i.e.,  translation  and  rotation),  elapsed  time  (pulse  code  modulated),  and  experi- 
memer-induced  event  markers.  Areas  of  interest  for  the  analysis  work  include  an  appraisal  of  indi¬ 
vidual  strategies  and  their  relation  to  task  success,  across-condition  comparisons  of  control 
characteristics  (e.g.,  when  stereo  vision  or  predictor  displays  are  used  in  the  experimental  designs), 
error  history  monitoring  (i.e.,  analysis  of  “lead-up”  events),  analysis  of  discrete  event  changes  (e.g., 
the  effects  of  discrete  time  delay  changes,  or  selections  of  new  robot  control  frames),  and  possibly  an 
appraisal  of  the  effects  of  controller  “physical”  characteristics  (e.g.,  axis  cross-talk,  deadband,  break¬ 
out  forces).  Finally,  there  are  a  number  of  outstanding  questions  to  be  addressed  with  regard  to  the 
use  and  development  of  the  data  logging  system  described  above.  For  instance,  is  it  desirable  and 
even  possible  to  use  the  data  to  generate  metrics  of  performance  suitable  for  complementing  experi¬ 
mental  and  subjective  results?  Do  other  forms  of  control  data  (other  than  onset,  direction  and  dura¬ 
tion  of  control  inputs)  exist  that  are  appropriate  for  logging?  How  can  the  concept  be  extended  to 
record  performance -related  parameters  other  than  control  inputs  (such  as  visual  search,  attention, 
mental  workload,  etc.)?  Can  a  data  logging  system  provide  a  useful  insight  into  the  operational  cost 
of  remote  space  operations?  Answers  to  these  questions  will  become  clearer,  following  the  use  of  the 
logging  system  during  the  experimental  phase  of  the  third  contract,  scheduled  for  the  Summer  of 
1990. 
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VERDEX:  A  VIRTUAL  ENVIRONMENT  DEMONSTRATOR  FOR  REMOTE 

DRIVING  APPLICATIONS 


Robert  J.  Stone 

The  National  Advanced  Robotics  Research  Centre 
Salford,  United  Kingdom 


The  National  Advanced  Robotics  Centre  was  founded  in  June  1988,  by  the  United  Kingdom’s 
Department  of  Trade  and  Industry  and  1 1  industrial  and  academic  shareholding  organisations.  The 
Centre  acts  as  a  focal  point  for  the  development  of  the  core  technologies  which  facilitate  the  next 
generation  of  robots,  and  is  geared  toward  projecting  the  UK  to  the  front  of  this  international  arena. 
One  of  the  key  areas  of  the  Centre’s  enabling  technologies  research  programme  is  that  of  the  human- 
system  interface,  phase  1  of  which  started  in  July  1989  and  is  currently  addressing  the  potential  of 
virtual  environments  to  permit  intuitive  and  natural  interactions  between  a  human  operator  and  a 
remote  robotic  vehicle.  The  aim  of  the  first  12  months  of  this  programme  (to  September,  1990)  is  to 
develop  a  virtual  iiuman-interface  demonstrator  for  use  later  as  a  test  bed  for  human  factors  experi¬ 
mentation.  This  presentation  will  describe  the  cunent  state  of  development  of  the  test  bed,  and  will 
outline  some  human  factors  issues  and  problems  for  more  general  discussion. 

In  brief,  the  virtual  telepresence  system  for  remote  driving  has  been  designed  to  take  the  follow¬ 
ing  form.  The  human  operator  will  be  provided  with  a  helmet-mounted  stereo  display  assi-mbly, 
facilities  for  speech  recognition  and  synthesis  (using  the  Marconi  Macrospeak  system),  and  a  VPL 
DataGlove  Model  2  unit.  The  vehicle  to  be  used  for  the  purposes  of  remote  driving  is  a  Cybermotion 
Navmaster  K2A  system,  which  will  be  equipped  with  a  stereo  camera  and  microphone  pair,  mounted 
on  a  motorised  high-speed  pan-and-tilt  head  incorporating  a  closed-loop  laser  ranging  sensor  for 
camera  convergence  control  (currently  under  contractual  development).  It  will  be  possible  to  relay 
information  to  and  from  the  vehicle  and  sensory  system  via  an  umbilical  or  RF  link.  The  aim  is  to 
develop  an  interactive  audio-visual  display  system  capable  of  presenting  combined  stereo  TV  pic¬ 
tures  and  virtual  graphics  windows,  the  latter  featuring  control  representations  appropriate  for  vehi¬ 
cle  driving  and  interaction  using  a  graphical  “hand,”  slaved  to  the  flex  and  tracking  sensors  of  the 
DataGlove  and  an  additional  helmet-mounted  Polhemus  IsoTrack  sensor. 

Developments  planned  for  the  virtual  environment  test  bed  include  transfer  of  operator  control 
between  remote  driving  and  remote  manipulation,  dextrous  end  effector  integration,  virtual  force  and 
tactile  sensing  (also  the  focus  of  a  current  ARRL  contract,  initially  employing  a  14-pneumatic  blad¬ 
der  glove  attachment),  and  sensor-driven  world  modelling  for  total  virtual  environment  generation 
and  operator-assistance  in  remote  scene  interrogation. 
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DESTABILIZING  EFFECTS  OF  VISUAL  ENVIRONMENT  MOTIONS 
SIMULATING  EYE  MOVEMENTS  OR  HEAD  MOVEMENTS 


K.  D.  White,  D.  Shuman,'  J.  H.  Kraniz,-  C.  B.  Woods,  and  L.  A.  Kuntz 

University  of  Florida 
Gainesville,  Florida 


Teleoperators  are  the  humans  who  control  devices  from  a  “distance.”  This  distance  might  be 
extreme  physical  separation,  as  in  remote  assembly  of  Space  Station  Freedom  by  ground-based  per¬ 
sonnel.  But  the  distance  could  be  effective  distance  of  scale,  for  the  manipulation  of  microscopic 
structures  like  single  living  cells  or  the  components  within  an  integiated  circuit  chip,  or  even  the 
conceptual  distance  of  safety,  rt  hcre  the  devices  effect  their  actions  inside  a  nuclear  reactor’s  core  or 
in  the  ocean’s  depths.  Virtual  environments  refer  to  the  synthesized  realities  that  can  be  generated  by 
various  types  of  computerized  displays,  not  only  visual  displays  but  also  acoustic,  tactile,  and  force- 
reflective  displays.  Video  games  are  common  but  usually  limited  examples  of  virtual  worlds.  An 
implicit  theme  is  telepresence,  a  term  whose  definition  is  imprecise.  It  involves  the  use  of  virtual 
environments  to  improve  the  efficiency  of  teleoperators  by  giving  them  a  compelling  sense  of  “being 
where  the  action  is.”  Telepresence  operations  seem  more  natural  and  facile,  and  thus  more  easily 
trained,  than  other  possible  models  for  the  human-computer  interface.  Consider  a  complex  assembly 
task,  controlled  by  wearing  a  sensor  garment  that  enslaves  a  robotic  arm  to  mimic  the  user’s  own 
arm  movements,  and  visualized  on  video  and  felt  by  tactile  and  force-reflective  feedbacks;  versus,  its 
control  by  typing  code  into  a  keyboard  with  feedback  via  numerical  tables.  Telepresence  seems  to 
endow  the  user  with  such  a  robust  mental  model  that  he  or  she  becomes  absorbed  into  the  synthetic 
reality  as  though  absorbed  into  a  vivid  dream. 

The  primary  challenge  for  reduction  to  practice  is  to  develop  a  paradigm  for  human-computer 
interaction  that  will  enable  telepresence  to  be  implemented  reliably.  The  natural  movements  of  the 
human  operator,  such  as  the  ability  to  “look  around”  in  the  virtual  world,  provide  an  important  cora- 
{xinent  for  this  paradigm  inasmuch  as  the  consistency  of  the  virtual  world  can  be  enhanced  by  its 
enslavement  to  sensed  head  or  eye  movements  of  the  human  user. 

In  the  present  paper,  we  explore  effects  on  the  human  of  exposure  to  a  visual  virtual 
environment  which  has  been  enslaved  to  simulate  the  human  user’s  head  movements  or  eye 
movements.  Specifically,  we  have  studied  the  capacity  of  our  experimental  subjects  to  maintain 
stable  spatial  orientation  in  the  context  of  moving  their  entire  visible  surroundings  by  using  the 
parameters  of  the  subjects’  natural  movements.  Our  index  of  the  subjects’  spatial  orientation  was  the 
extent  of  involuntary  sways  of  the  body  while  attempting  to  stand  still,  as  measured  by  translations 
and  rotations  of  the  head.  We  also  observed,  informally,  their  symptoms  of  motion  sickness. 


'Now  at  Agricultural  Research  Service,  U.S.  Department  of  Agriculture. 
-Now  at  Honeywell  Phoenix  Techology  Center. 


148 


METHODS 


Subjects 

A  total  of  93  university  students  or  staff,  50  males  and  43  females  ranging  in  age  from  18  to 
43  years  old,  participated  as  unpaid  volunteers  in  one  of  four  experiments.  Some  students  received 
course  credit.  Subjects  were  prescreened  for  medical  histories  (self-report)  and  any  with  possible 
vestibular  defects  were  excused.  Any  subjects  requiring  refractive  correction  used  their  own  pre¬ 
scribed  lenses. 

Apparatus  and  Procedures 

Head  Position  Sensing.  An  acoustic  timing  principle  was  used  to  locate  in  space  the  headgear 
worn  by  each  subject.  This  lightweight  headgear  contained  two  click  sources,  the  wavefronts  from 
which  were  detected  with  microsec  timing  resolution  at  each  of  four  microphones  attached  to  the 
ceiling  of  the  testing  room.  Software  algorithms  for  windowing  the  microphone  outputs  and  per¬ 
forming  redundancy  checks,  then  implementing  solutions  of  the  Pythagorean  theorem,  permitted 
each  click  source  to  be  located  in  space  with  translation  precision  below  1mm  at  a  sampling  rate  of 
50  Hz  maximum.  Resolution  of  the  rotation  angle  for  the  line  connecting  the  two  click  sources  was 
below  0.2  deg  (25  Hz  maximum  sampling  rate).  The  headgear  configuration  did  not  permit  resolu¬ 
tion  of  rotations  around  the  line  connecting  the  click  sources  (pitch),  but  did  permit  resolution  of  the 
other  five  degrees  of  freedom  for  head  movements. 

Visual  Environment.  Each  subject  stood  near  the  axis  of  a  vertical  hemicylindrical  screen  onto 
which  was  projected  a  pattern  of  vertical  stripes.  This  pattern  subtended  in  excess  of  180  deg  of 
azimuth  and  120  deg  in  elevation.  The  instantaneous  azimuth  positioning  of  the  pattern  was  under 
software  control  with  a  resolution  of  approximately  0.04  deg  at  a  maximum  update  rate  exceeding 
1  KHz.  Impedances  within  the  projection  system  were  compensated  for  via  nonlinear  control  signals 
when  motions  simulating  saccadic  eye  movements  were  employed.  In  those  particular  experiments, 
the  control  signals  were  precomputed  to  reduce  lag  time  to  below  1  msec.  When  motions  enslaved  to 
the  subject’s  head  movements  were  employed,  it  was  impractical  to  precompute  the  control  signals. 
Thus,  there  was  an  additional  lag  time  for  computation  of  about  150  msec  and  the  update  rate  was 
reduced  to  10  Hz. 

Simulations  of  Saccadic  Eye  Movements.  Two  small  fixation  lights  rear-projected  onto  the  sur¬ 
rounding  screen  could  be  alternated  to  direct  horizontal  saccades  of  particular  azimuth  extent  (0.5, 1, 
2,  3,  4,  6,  8,  or  16  deg).  Motions  of  the  pattern  front-projected  onto  the  screen  had  characteristics  of 
average  vf’/iocity  and  motion  duration  similar  to  those  for  each  corresponding  saccade  extent.^  In  two 
experiments,  the  saccade-like  stimulus  motions  occurred  independent  of  the  subject’s  eye  move¬ 
ments.  In  another  experiment,  the  stimulus  motions  were  triggered  by  subject  eye  velocity,  derived 
from  the  electro-oculogram,  when  the  eye  had  moved  15-20  aremin  from  its  initial  fixation  during  a 
guided  voluntary  saccade. 


^A.  T.  Bahill,  M.  R.  Clark  &  L.  Stark,  Mathematical  Biosciences,  24,  191-204  (1975). 

Partially  supported  by  a  grant  from  the  National  Eye  Institute  to  KDW  (R23  EY03640),  and  by  an  NSF  graduate 
fellowship  to  JHK. 
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Head  Movement  Feedback  Conditions.  Each  sample  of  the  head’s  fore/aft  and  left/right 
translations  plus  head  yaw  angle  permitted  real  time  calculation  of  an  equivalent  azimuth  change  for 
the  surrounding  visual  stimulus.  This  equivalent  azimuth,  modified  by  a  gain  factor  that  was  the 
parameter  selected  for  study,  controlled  motion  of  the  visual  stimulus  in  this  experiment.  The  gain 
factors  used  were  +2,  +1.5,  +1,  +0.5, 0,  -0.5,  -1,  -1.5,  and  -2.  The  gain  of +1,  for  example,  could 
serve  in  principle  to  keep  the  stimulu.''  ‘straight  ahead”  by  its  faithfully  tracking  the  head’s 
movements.  Larger  gains  moved  the  stimulus  farther  and  faster  than  would  accurate  head  tracking; 
smaller  gains  moved  the  stimulus  less  far  and  more  slowly.  2^ro  gain  made  the  stimulus  remain  still 
as  in  the  natural  environment.  Negative  gains  reversed  the  left/right  correspondence  between  head 
movements  and  stimulus  motions. 

Data  Analysis 

Control  Conditions.  Although  the  stability  of  a  standing  human  has  been  modelled  as  an 
inverted  pendulum,  since  the  body’s  center  of  gravity  is  above  its  support  base,  the  problem  with 
such  a  model  is  that  the  human  body  is  not  mechanically  rigid.  Rather,  it  is  a  jointed  mechanism 
controlled  by  over  400  muscles  and  several  sensory  feedback  loops,  with  correspondingly  complex 
dynamics.  Furthermore,  people  grow  to  various  heights,  weights  and  distributions  of  body  mass; 
with  various  extent  of  conditioning  for  their  musculature;  and  with  varied  skills  of  sensorimotor 
integration.  Individual  variations  in  the  body’s  dynamics  are,  therefore,  prominent  and,  as  a 
consequence,  we  rely  on  the  use  of  experimental  designs  allowing  each  individual  to  serve  as  his/her 
own  control. 

Also,  it  is  impractical  to  open  the  feedback  loops  from  proprioceptive/kinesthetic  or  vestibular 
senses  but  it  is  relatively  easy  to  alter  those  loops  by  asking  the  subject  to  stand  only  on  one  foot  or 
else  on  a  compliant  surface.  As  reported  in  the  results  below,  there  are  multiple  techniques  for  con¬ 
trolling  the  visual  feedback  loop  within  each  subject,  including:  (a)  injecting  noise  via  random 
motions  of  the  visual  environment,  (b)  stabilizing  the  environment  via  motion  feedbacK  from  head 
movements,  (c)  eliminating  contour  from  the  visual  environment,  or  (d)  eliminating  visibility  by 
turning  off  the  lights  and  (e)  by  closing  the  eyes. 

Within-Subject  Figure  of  Merit  for  Body  Stability.  There  is  no  consensus  in  the  literature  for  a 
representative  figure  of  merit  to  describe  such  complicated  behavioral  dynamics  as  those  under  con¬ 
sideration.  We  have  not  been  satisfied,  in  general,  with  time-domain  analyses  such  as  average 
movement  velocity  or  RMS  position  error,  since  these  statistical  summaries  tend  to  obscure  details 
of  the  responses  which  we  find  to  be  correlated  with  experimental  manipulations.  We  have,  however, 
found  one  particular  form  of  time-domain  analysis  to  be  useful  in  certain  circumstances,  and  that 
form  is  time-locked  response  averaging.  We  plan  to  explore  in  future  work  the  more  general  utility 
of  a  “chaotic”  time-domain  (phase-sensitive)  analysis. 

We  have  enjoyed  considerable  success  in  accounting  for  individual  variations  among  subjects  by 
using  frequency-domain  analysis,  however.  Specifically,  each  subject  is  measured:  (a)  under  control 
conditions  to  determine  his/her  individual  power  spectral  density  (PSD)  functions  for  body  move¬ 
ments  over  time,  and  (b)  under  experimental  conditions,  to  determine  any  changes  in  the  PSD 
functions  induced  experimentally.  The  ratio  between  PSD  functions  (a)  and  (b)  is  the  gain,  as  a 
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function  of  the  frequency,  induced  by  the  experimental  manipulation.  We  find  these  gains  to  be 
relatively  consistent  betwen  individuals  (standard  errors  of  measurement  typically  less  than  1  dB). 


RESULTS 


Figure  1  shows  sample  records  of  two  subjects’  lateral  head  movements  while  viewing  moving 
surroundings  (middle  two  traces).  For  the  top  trace,  the  motion  of  the  stimulus  was  saccade-like, 
while  for  the  bottom  trace,  the  stimulus  tracked  the  subject’s  head  movements  (gain  =  +1.5).  These 
sample  records  have  been  displaced  vertically  for  clarity. 

The  top  trace  of  Figure  1  shows  typical  lateral  sways  made  by  subject  S  1  when  the  entire  visible 
surroundings  suddenly  moved  with  the  velocity  (66  deg/sec)  and  duration  (30  msec)  characteristic  of 
a  2  deg  saccadic  eye  movement.  Note  the  relatively  pronounced  fluctuations  in  head  position.  By  a 
comparison  against  the  second  trace,  S  I’s  lateral  sways  in  still  surroundings,  it  is  clear  that  the  stim¬ 
ulus  motion  exerts  a  strong  destabilizing  effect. 

The  bottom  trace  of  Figure  1  shows  lateral  sways  made  by  subject  S  2  while  the  azimuth  position 
of  the  visible  stimulus  tracked  S  2’s  own  head  movements.  By  comparison  against  the  third  trace, 
subject  S  2’s  lateral  sways  in  still  surroundings,  the  destabilizing  effect  of  head  tracking  is  apparent. 
Individual  variations  in  stability  can  be  noted  by  comparing  the  third  trace  (S  2)  against  the  second 
trace  (S  1),  sways  made  in  still  surroundings. 

Figure  2  depicts  averaged  yaw  movements  (t.e.,  head  rotations)  made  in  reponse  to  saccade-like 
motions  of  the  stimulus,  plotted  as  a  function  of  time  after  the  motion  began.  Each  plotted  point  is 
based  on  10  repetitions  per  subject  times  18  subjects,  or  the  average  of  180  measures  of  head  yaw 
angle  per  condition.  The  measures  were  carried  out  in  three  conditions:  (1)  saccade-only,  the  subject 


Figure  1 .  Sample  records  of  head  movements  in  still  vs.  moving  environments  (saccade-like  or  head 
tracking  motions),  for  two  subjects. 
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Figure  2.  Response  averages  showing  head  yaw  elicited  by  saccade-like  stimulus  motions.  Motions 
triggered  by  saccades  are  less  effective  than  untriggered  motions  for  eliciting  these  responses. 

moved  their  eyes  as  directed  by  fixation  lights  but  there  was  no  additional  motion  of  the  stimulus, 

(2)  motion  triggered  by  the  saccade,  in  which  a  saccade-like  stimulus  motion  took  place  while  the 
subject’s  eyes  were  moving  in  a  saccade,  and  (3)  untriggered  motion,  in  which  a  saccade-like 
stimulus  motion  briefly  preceded  (<100  msec)  the  subject’s  execution  of  a  saccadic  eye  movement. 
In  the  saccade-only  condition,  subjects  showed  a  slight  tendency  to  yaw  the  head  as  well  as  to  move 
the  eyes.  Average  head  yaw  for  saccade-only  is  taken  as  the  baseline  in  this  plot.  This  baseline 
motion  is  subtracted  from  the  head  yaws  in  response  to  stimulus  motions. 

It  is  clear  in  Hgure  2  that  untriggered  saccade-like  stimulus  motions  elicit  a  small  but  reliable 
yaw  (the  peak  at  0.5  sec  represents  10  standard  errors  difference).  This  may  be  qualitatively  similar 
to  the  lateral  head  translations  illustrated  in  the  top  trace  of  Figure  1 .  However,  Figure  2  also  shows 
that  the  effect  of  the  saccade-like  stimulus  motions  can  be  lessened  dramatically  by  causing  them  to 
coincide  with  the  subject’s  own  voluntary  saccades.  In  this  latter  case,  possibly  a  signal  generated 
internally  by  the  subject’s  oculomotor  system  (viz.,  corollary  discharge  or  efference  copy)  alters  the 
consequences  that  otherwise  accompany  the  stimulus  motion. 

Figure  3  is  a  plot  derived  from  fast  Fourier  transforms  of  sway  records  analogous  to  those  shown 
in  Figure  1 .  For  the  plot  shown  here,  the  power  spectral  density  function  (PSD)  for  each  subject  was 
obtained  in  still  surroundings  and  this  was  taken  as  the  baseline  for  that  individual.  PSDs  for  that 
subject  under  other  conditions  of  stimulation  were  divided  by  his/her  own  baseline  PSD.  This 
resulted  in  gain  profrles  (dB  relative  to  baseline)  as  a  function  of  sway  frequency  (Hz).  Each  point  on 
Figure  3  represents  the  average  gain  across  24  subjects  (18  for  saccades  condition)  within  a  fre¬ 
quency  band  0.5  Hz  wide.  Typical  standard  errors  associated  with  each  point  are  under  1  dB. 

The  six  curves  shown  in  Figure  3  depict  the  effects  on  subjects’  stability  of  (1)  closing  their  eyes, 
(2)  eliminating  visible  contour  (subjects  wore  diffusing  goggles),  (3)  eliminating  illumination 
(subjects  wore  opaque  goggles,  (4)  moving  the  azimuth  of  the  surroundings  to  track  the  subject’s 
head  movements  (gain  ±1.0),  (5)  moving  the  azimuth  of  the  surroundings  randomly,  and  (6)  the 
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Figure  3.  Comparisons  of  the  gain  in  lateral  sway  as  a  function  of  sway  frequency,  for  experimental 
and  control  condition. 

subjects  made  directed  saccadic  eye  movements  across  the  still  surroundings.  Two  of  the  curves  on 
Figure  3  were  moved  vertically  to  reduce  overlap:  the  “lights  out”  curve  was  moved  down  2  dB  and 
the  “head  track”  curve  was  moved  down  3  dB.  “Eyes  closed”  caused  significantly  larger  sways  than 
either  “no  contours”  or  “lights  out,”  which  did  not  differ  reliably.  “Head  track”  resulted  in  signifi¬ 
cantly  smaller  sways  than  in  those  conditions. 

All  six  curves  show  a  general  trend  of  decreasing  gain  with  increasing  fiequency,  particularly  so 
below  2  Hz.  To  a  first  apjxoximation,  the  gain  prr^le  associated  with  any  of  these  six  conditions 
appears  nearly  interchangeable  with  the  profile  f.  a  another  condition  by  simply  shifting  the  curve 
up  or  down  on  the  graph.  Since  all  of  these  conditions  might  serve  to  reduce  or  degrade  visual  feed¬ 
back  from  the  environment,  this  Hrst  approximation  similarity  is  not  overly  surprising.  In  detail, 
however,  it  is  interesting  to  note  that  the  saccades  condition  (6)  actually  made  Ae  subjects  sway  less 
than  did  fixating  in  still  surroundings.  Also  note  that  the  head  track  condition  (4)  is  the  only  curve 
with  a  large  gain  decrease  at  0.75  Hz  (midpoint  for  the  0.5  to  1.0  Hz  band).  The  frequency  depen¬ 
dence  of  gain  profiles  with  head  tracking  is  more  apparent  in  the  following  graphs. 

Figures  4  and  5  compare  the  effects  of  four  different  gain  factors  for  head  tracking.  The  gain 
factor  parameter  relates  azimuth  positioning  of  the  visible  surround  to  the  combined  fore-aft  and  left- 
right  translations  plus  yaw  angle  sensed  for  the  head.  Each  point  on  these  figures  represents  the  aver¬ 
age  movement  gain  across  33  subjects,  calculated  within  0.5  Hz  bandwidths.  These  gains,  the  ratios 
of  PSD  profiles  in  tracking  vs.  baseline  conditions,  were  referenced  to  “random  motion”  baseline 
measures  with  corresponding  gain  factors.  These  baselines  were  necessary  because  the  visible  sur¬ 
round  exhibited  a  slight  random  jitter  due  to  system  noise  even  when  attempting  to  track  a  fixed  arti¬ 
ficial  head. 

There  are  two  noteworthy  features  shown  in  Figures  4  and  5.  First,  the  gain  profiles  are  clearly 
not  flat  but  instead  rise  or  fall  in  a  frequency-dependent  manner.  Thus,  we  must  conclude  that  mov¬ 
ing  the  visual  stimulus  by  tracking  the  subject’s  head  differs  qualitatively  from  the  various  other 
forms  of  altered  visual  feedback  depicted  in  Figure  3,  because  those  manipulations  have  yielded 
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Figure  4.  Lateral  sway  gain  as  a  function  of  frequency  when  the  azimuth  of  the  surrounds  tracked  the 
subject’s  head  movements.  The  four  curves  represent  different  gain  factor. 


Yow  Frequency  (Hz) 


Figure  5.  Yaw  movement  gain  as  a  function  of  frequency,  analogous  to  Figure  4. 

results  that  are  highly  similar  across  frequency.  Second,  whether  each  curve  in  Figures  4  and  5  rises 
or  falls  in  each  frequency  band  is  highly  dependent  on  the  sign  of  the  gain  factor  parameter.  For  both 
lateral  sway  (Fig.  4)  and  yaw  movements  (Fig.  5),  a  gain  peak  associated  with  positive  parameter 
values  becomes  a  gain  trough  when  associated  with  negative  parameter  values,  and  vice  versa. 

Even  though  subjects  in  the  head  tracking  experiment  showed  greater  stability  than  did  subjects 
who  simply  closed  their  eyes,  10  out  of  the  33  reported  symptoms  of  motion  sickness.  For  4  of  those 

subjects,  symptoms  persisted  more  than  an  hour,  and  for  one  subject,  persisted  5  hours.  Another 
8  subjects  found  the  participation  unpleasant  but  not  nauseogenic.  In  contrast,  3  subjects  reported 
extreme  exhilaration  apparently  akin  to  taking  an  amusement  park  ride.  Only  one  subject  guessed 
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that  the  stimulus  tracked  her  head  movements,  while  another  subject  guessed  that  her  head  had  been 
captured  by  motion  of  the  visual  stimulus. 


CONCLUSIONS 


We  find  that  exposure  to  a  visual  virtual  environment  enslaved  to  simulate  a  subject’s  own  eye 
movements  or  head  movements  can  have  deleterious  effects  on  the  subject’s  spatial  stability.  When 
the  entire  visible  surroundings  were  moved  with  the  same  parameters  of  angular  velocity  and  motion 
duration  which  characterize  natural  saccadic  eye  movements,  subjects  tended  to  lose  lateral  stability 
(Fig.  1)  and  to  make  head  yaw  movements  (Fig.  2,  untriggered  motions).  This  pattern  of  response 
was  cancelled,  however,  when  saccade-like  stimulus  motions  were  made  to  coincide  with  the 
subject’s  own  voluntary  eye  movements  (also  in  Fig.  2,  triggered  by  saccade).  It  seems  plausible  to 
interpret  these  findings  in  the  light  if  theoretical  frameworks  which  propose  that  a  subject  generates  a 
hypothetical  internal  signal  regarding  their  own  voluntary  eye  movements.  Such  an  internal  signal 
(corollary  discharge  or  efference  copy)  would  presumably  serve  in  the  natural  world  to  mitigate  the 
effects  of  retinal  image  motion  accompanying  one’s  voluntary  eye  movement.  In  the  present  virtual 
world,  the  same  hypothetical  internal  signal  presumably  mitigates  the  effects  of  external  stimulus 
motion  if  it  is  made  at  the  same  time  when  this  internal  signal  is  active. 

The  effects  on  a  subject’s  spatial  stability  when  the  visual  surroundings  tracked  head  movements 
are  less  readily  explained.  First,  any  subject  intentions  to  move  the  head  were  discouraged,  inasmuch 
as  (a)  they  were  instructed  to  stand  still  and  (b)  they  became  significantly  at  risk  of  falling  if  they 
moved  the  head  voluntarily.  It  must  be  questioned  whether  an  efference  copy  or  corollary  discharge 
internal  signal  could  exist  without  subjects’  internal  plans  to  make  voluntary  movements.  Second, 
subjects’  movement  gain  profiles  in  Figures  4  and  5  are  reminiscent  of  the  changes  in  poles  and 
zeros  in  the  complex  domain  when  a  device  receiving  feedback  undergoes  a  change  in  its  feedback’s 
sign.  It  is  tempting  to  speculate  that  head  tracking  conditions  effectively  reveal  some  of  the  subjects’ 
own  internal  feedback  mechanisms.  Unfortunately,  the  lag  time  required  to  update  stimulus  azimuth 
after  sensing  head  position  and  orientation  (150  msec)  was  sufficient  to  complicate  detailed  interpre¬ 
tations  along  these  lines.  Third,  it  may  be  important  to  understand  why  so  many  subjects  in  the  head 
tracking  experiments  experienced  motion  sickness  symptoms  or  other  discomforts,  even  though  they 
did  not  become  very  greatly  disoriented.  We  conjecture  that  the  head-tracking  visual  feedback  must 
have  been  in  conflict  with  the  subjects’  own  internal  models  of  their  orientation  in  space.  Such 
models  are  probably  multisensory,  though  highly  dependent  on  visual  feedback,  and  probably  estab¬ 
lished  below  the  level  of  one’s  conscious  awareness. 

We  believe  that  the  design  of  virtual  environments  which  are  enslaved  to  the  eye  or  head  move¬ 
ments  of  a  teleoperator  must  take  into  account  the  effects  on  the  human.  In  the  present  experiments, 
we  found  that  these  effects  can  include  loss  of  stability,  motion  sickness,  or  other  undesirable  influ¬ 
ences  on  situational  awareness.  These  presumably  could  be  mitigated  by  carefully  matching  sensor 
resolutions  and  real  time  controls  of  the  virtual  environment  to  the  capacities  of  the  human 
participants. 
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